[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:19 dist-test-slave-rmhg krb5kdc[15705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:19 dist-test-slave-rmhg krb5kdc[15705](info): set up 2 sockets
Aug 10 14:40:19 dist-test-slave-rmhg krb5kdc[15705](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 14:40:20.617834 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:40:20 dist-test-slave-rmhg krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:40:20.749369 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:35913
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:35913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:20.868783 15726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:20.869115 15726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:20.869204 15726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:20.873335 15726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:20.873412 15726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:20.873438 15726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:20.873459 15726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:20.873479 15726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:20.879112 15726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:35913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:35913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:20.880354 15726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:20.881500 15726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:20.891415 15740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:20.891364 15726 server_base.cc:1034] running on GCE node
W20260810 14:40:20.891384 15736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:20.891389 15737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:20.892338 15726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:20.893710 15726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:20.894894 15726 hybrid_clock.cc:648] HybridClock initialized: now 1786372820894877 us; error 111 us; skew 500 ppm
Aug 10 14:40:20 dist-test-slave-rmhg krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372820, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:20.898007 15726 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:40:20.899078 15726 webserver.cc:460] Webserver started at http://127.15.84.62:37033/ using document root <none> and password file <none>
I20260810 14:40:20.899724 15726 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:20.899796 15726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:20.900018 15726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:20.902451 15726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "2062e0c082384cd187538717b495c773"
format_stamp: "Formatted at 2026-08-10 14:40:20 on dist-test-slave-rmhg"
server_key: "239ed4844d1e02e2885436ab753f2b49"
server_key_iv: "3973e891cb157a4a40b6fbf90b6fb36e"
server_key_version: "encryptionkey@0"
I20260810 14:40:20.903086 15726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "2062e0c082384cd187538717b495c773"
format_stamp: "Formatted at 2026-08-10 14:40:20 on dist-test-slave-rmhg"
server_key: "239ed4844d1e02e2885436ab753f2b49"
server_key_iv: "3973e891cb157a4a40b6fbf90b6fb36e"
server_key_version: "encryptionkey@0"
I20260810 14:40:20.907334 15726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:40:20.910195 15752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:20.911643 15726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:20.911787 15726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "2062e0c082384cd187538717b495c773"
format_stamp: "Formatted at 2026-08-10 14:40:20 on dist-test-slave-rmhg"
server_key: "239ed4844d1e02e2885436ab753f2b49"
server_key_iv: "3973e891cb157a4a40b6fbf90b6fb36e"
server_key_version: "encryptionkey@0"
I20260810 14:40:20.911906 15726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:20.924373 15726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:20.927420 15726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:20.927610 15726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:20.934986 15726 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:35913
I20260810 14:40:20.935021 15826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:35913 every 8 connection(s)
I20260810 14:40:20.936038 15726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:20.937741 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 15726
I20260810 14:40:20.937922 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:20.939714 15696 external_mini_cluster.cc:1428] Setting key 09b4feae673428c8a27e1c815f150163
I20260810 14:40:20.939752 15828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:20.946815 15828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773: Bootstrap starting.
I20260810 14:40:20.949733 15828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:20.950770 15828 log.cc:824] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:20.953097 15828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773: No bootstrap required, opened a new log
I20260810 14:40:20.957434 15828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062e0c082384cd187538717b495c773" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 35913 } }
I20260810 14:40:20.957716 15828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:20.957760 15828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2062e0c082384cd187538717b495c773, State: Initialized, Role: FOLLOWER
I20260810 14:40:20.958294 15828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062e0c082384cd187538717b495c773" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 35913 } }
I20260810 14:40:20.958443 15828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:20.958508 15828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:20.958618 15828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:20.959640 15828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062e0c082384cd187538717b495c773" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 35913 } }
Aug 10 14:40:20 dist-test-slave-rmhg krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:20.960021 15828 leader_election.cc:302] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2062e0c082384cd187538717b495c773; no voters:
I20260810 14:40:20.960425 15828 leader_election.cc:288] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:20.960527 15836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:20.960723 15836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [term 1 LEADER]: Becoming Leader. State: Replica: 2062e0c082384cd187538717b495c773, State: Running, Role: LEADER
I20260810 14:40:20.961078 15836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062e0c082384cd187538717b495c773" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 35913 } }
I20260810 14:40:20.961958 15828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:20.963025 15839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2062e0c082384cd187538717b495c773. Latest consensus state: current_term: 1 leader_uuid: "2062e0c082384cd187538717b495c773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062e0c082384cd187538717b495c773" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 35913 } } }
I20260810 14:40:20.963173 15839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:20.963469 15837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2062e0c082384cd187538717b495c773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062e0c082384cd187538717b495c773" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 35913 } } }
I20260810 14:40:20.963615 15837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:20.964493 15850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:20.967686 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:20.943775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.15.84.62:35913)
0810 14:40:20.944333 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:20.944355 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:20.945585 (+ 1230us) server_negotiation.cc:407] Connection header received
0810 14:40:20.949119 (+ 3534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:20.949146 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:20.949686 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:20.950125 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:20.952638 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:20.954173 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:20.955655 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:20.956460 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:20.956662 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:20.961201 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:20.961210 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:20.961229 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:20.961274 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:20.963354 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:20.964106 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:20.964114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:20.964120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:20.964284 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:20.964756 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:20.964760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:20.964762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:20.965469 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:20.965687 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:20.966366 (+ 679us) server_negotiation.cc:299] Negotiation successful
0810 14:40:20.966670 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":169,"threads_started":1}
I20260810 14:40:20.968035 15850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:20.974591 15850 catalog_manager.cc:1347] Generated new cluster ID: 1eae9dbb859b49869f15d457ec49c07b
I20260810 14:40:20.974696 15850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:20.989099 15850 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:20.990523 15850 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:21.005170 15850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2062e0c082384cd187538717b495c773: Generated new TSK 0
I20260810 14:40:21.006187 15850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:40:21.040117 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:35913
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:21.162132 15868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:21.162560 15868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:21.162631 15868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:21.166992 15868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:21.167071 15868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:21.167141 15868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:40:21.172843 15868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:35913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:21.174160 15868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:21.175290 15868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:21.185653 15875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:21.185665 15878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:21.185686 15876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:21.185802 15868 server_base.cc:1034] running on GCE node
I20260810 14:40:21.186408 15868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:21.187058 15868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:21.188220 15868 hybrid_clock.cc:648] HybridClock initialized: now 1786372821188194 us; error 48 us; skew 500 ppm
Aug 10 14:40:21 dist-test-slave-rmhg krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372821, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:21.191061 15868 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:40:21.192020 15868 webserver.cc:460] Webserver started at http://127.15.84.1:43131/ using document root <none> and password file <none>
I20260810 14:40:21.192598 15868 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:21.192677 15868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:21.192901 15868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:21.195143 15868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "52cbe7bb9f2e448bae752a44ea23b3f3"
format_stamp: "Formatted at 2026-08-10 14:40:21 on dist-test-slave-rmhg"
server_key: "4614a00eada301fbddbb248e6a9ff571"
server_key_iv: "bb738c203c5dfde3cde206c90abf7213"
server_key_version: "encryptionkey@0"
I20260810 14:40:21.195739 15868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "52cbe7bb9f2e448bae752a44ea23b3f3"
format_stamp: "Formatted at 2026-08-10 14:40:21 on dist-test-slave-rmhg"
server_key: "4614a00eada301fbddbb248e6a9ff571"
server_key_iv: "bb738c203c5dfde3cde206c90abf7213"
server_key_version: "encryptionkey@0"
I20260810 14:40:21.199820 15868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:40:21.202461 15889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:21.203769 15868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:40:21.203894 15868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "52cbe7bb9f2e448bae752a44ea23b3f3"
format_stamp: "Formatted at 2026-08-10 14:40:21 on dist-test-slave-rmhg"
server_key: "4614a00eada301fbddbb248e6a9ff571"
server_key_iv: "bb738c203c5dfde3cde206c90abf7213"
server_key_version: "encryptionkey@0"
I20260810 14:40:21.204005 15868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:21.222090 15868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:21.225528 15868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:21.225755 15868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:21.226401 15868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:21.227309 15868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:21.227360 15868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:21.227417 15868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:21.227447 15868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:21.236608 15868 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:38025
I20260810 14:40:21.236649 16063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:38025 every 8 connection(s)
I20260810 14:40:21.237680 15868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:40:21.238235 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 15868
I20260810 14:40:21.238354 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:40:21.238662 15696 external_mini_cluster.cc:1428] Setting key 6c3e8a2487892bd1f7910ea440b5df5b
Aug 10 14:40:21 dist-test-slave-rmhg krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372821, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
I20260810 14:40:21.255195 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:21.240017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:44943 (local address 127.15.84.62:35913)
0810 14:40:21.240167 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:21.240172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:21.241282 (+ 1110us) server_negotiation.cc:407] Connection header received
0810 14:40:21.242922 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:21.242926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:21.242988 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:21.243082 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:21.244812 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.245441 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.247316 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.247576 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.247676 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:21.250929 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:21.250933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:21.250935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:21.250968 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:21.252495 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:21.253115 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:21.253118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:21.253120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:21.253324 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:21.253797 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:21.253800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:21.253801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:21.254035 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:21.254144 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:21.254868 (+ 724us) server_negotiation.cc:299] Negotiation successful
0810 14:40:21.255016 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:40:21.256309 16074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:21.240537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:35913 (local address 127.15.84.1:44943)
0810 14:40:21.241087 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:21.241179 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:21.242663 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:21.243312 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:21.243320 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:21.243814 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:21.244596 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:21.244622 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.245599 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.245603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:21.247154 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:21.247170 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.247762 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.247767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:21.248017 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:21.248718 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:21.248743 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:21.250702 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:21.252650 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:21.252658 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:21.252680 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:21.252999 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:21.253428 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:21.253431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:21.253433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:21.253675 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:21.254163 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:21.254172 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:21.254570 (+ 398us) client_negotiation.cc:769] Sending connection context
0810 14:40:21.254820 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 14:40:21.255246 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":143,"threads_started":1}
I20260810 14:40:21.257633 16069 heartbeater.cc:347] Connected to a master server at 127.15.84.62:35913
I20260810 14:40:21.257929 16069 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:21.258570 16069 heartbeater.cc:511] Master 127.15.84.62:35913 requested a full tablet report, sending...
I20260810 14:40:21.260386 15776 ts_manager.cc:194] Registered new tserver with Master: 52cbe7bb9f2e448bae752a44ea23b3f3 (127.15.84.1:38025)
I20260810 14:40:21.261612 15776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:44943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:40:21.270336 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:35913
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:21.391963 16078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:21.392303 16078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:21.392357 16078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:21.396535 16078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:21.396629 16078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:21.396693 16078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:40:21.402290 16078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:35913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:21.403419 16078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:21.404452 16078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:21.415367 16088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:21.415370 16087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:21.415598 16078 server_base.cc:1034] running on GCE node
W20260810 14:40:21.415877 16090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:21.416334 16078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:21.417094 16078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:21.418315 16078 hybrid_clock.cc:648] HybridClock initialized: now 1786372821418277 us; error 49 us; skew 500 ppm
Aug 10 14:40:21 dist-test-slave-rmhg krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372821, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:21.421253 16078 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:40:21.422317 16078 webserver.cc:460] Webserver started at http://127.15.84.2:40513/ using document root <none> and password file <none>
I20260810 14:40:21.422940 16078 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:21.423018 16078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:21.423255 16078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:21.425642 16078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "0a9d64f16bdf439d97a23b0e4dbabbe1"
format_stamp: "Formatted at 2026-08-10 14:40:21 on dist-test-slave-rmhg"
server_key: "bd7f9aff290e50a062ca18b141831b8a"
server_key_iv: "28548dc74b518e66f08df10cdc7e112f"
server_key_version: "encryptionkey@0"
I20260810 14:40:21.426219 16078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "0a9d64f16bdf439d97a23b0e4dbabbe1"
format_stamp: "Formatted at 2026-08-10 14:40:21 on dist-test-slave-rmhg"
server_key: "bd7f9aff290e50a062ca18b141831b8a"
server_key_iv: "28548dc74b518e66f08df10cdc7e112f"
server_key_version: "encryptionkey@0"
I20260810 14:40:21.430392 16078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 14:40:21.433053 16100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:21.434423 16078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:21.434554 16078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "0a9d64f16bdf439d97a23b0e4dbabbe1"
format_stamp: "Formatted at 2026-08-10 14:40:21 on dist-test-slave-rmhg"
server_key: "bd7f9aff290e50a062ca18b141831b8a"
server_key_iv: "28548dc74b518e66f08df10cdc7e112f"
server_key_version: "encryptionkey@0"
I20260810 14:40:21.434695 16078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:21.449110 16078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:21.452369 16078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:21.452582 16078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:21.453326 16078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:21.454496 16078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:21.454557 16078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:21.454625 16078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:21.454661 16078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:21.464892 16078 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:43541
I20260810 14:40:21.464915 16286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:43541 every 8 connection(s)
I20260810 14:40:21.465991 16078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:40:21.468245 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 16078
I20260810 14:40:21.468351 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:40:21.468613 15696 external_mini_cluster.cc:1428] Setting key 9755b0d503247a8a48e0329b6ba931a0
Aug 10 14:40:21 dist-test-slave-rmhg krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372821, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
I20260810 14:40:21.481832 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:21.468061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:36259 (local address 127.15.84.62:35913)
0810 14:40:21.468186 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:21.468190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:21.469398 (+ 1208us) server_negotiation.cc:407] Connection header received
0810 14:40:21.470934 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:21.470938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:21.470992 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:21.471067 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:21.472487 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.473030 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.474718 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.474946 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.475026 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:21.478016 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:21.478020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:21.478022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:21.478050 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:21.479452 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:21.480063 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:21.480066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:21.480068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:21.480172 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:21.480549 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:21.480552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:21.480554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:21.480764 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:21.480854 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:21.481512 (+ 658us) server_negotiation.cc:299] Negotiation successful
0810 14:40:21.481659 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
I20260810 14:40:21.482945 16293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:21.468628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:35913 (local address 127.15.84.2:36259)
0810 14:40:21.469253 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:21.469305 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:21.470711 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:21.471275 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:21.471283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:21.471680 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:21.472313 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:21.472341 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.473200 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.473204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:21.474596 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:21.474609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.475126 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.475131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:21.475358 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:21.476006 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:21.476030 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:21.477814 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:21.479586 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:21.479601 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:21.479620 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:21.479905 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:21.480280 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:21.480283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:21.480285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:21.480445 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:21.480873 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:21.480880 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:21.481208 (+ 328us) client_negotiation.cc:769] Sending connection context
0810 14:40:21.481435 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 14:40:21.481868 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":161,"threads_started":1}
I20260810 14:40:21.484179 16288 heartbeater.cc:347] Connected to a master server at 127.15.84.62:35913
I20260810 14:40:21.484488 16288 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:21.485029 16288 heartbeater.cc:511] Master 127.15.84.62:35913 requested a full tablet report, sending...
I20260810 14:40:21.486238 15776 ts_manager.cc:194] Registered new tserver with Master: 0a9d64f16bdf439d97a23b0e4dbabbe1 (127.15.84.2:43541)
I20260810 14:40:21.487022 15776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:36259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:40:21.499595 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:35913
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:21.620090 16296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:21.620412 16296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:21.620460 16296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:21.624723 16296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:21.624806 16296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:21.624868 16296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:40:21.630506 16296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:35913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:21.631702 16296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:21.632786 16296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:21.642758 16310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:21.642762 16303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:21.642758 16304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:21.643193 16296 server_base.cc:1034] running on GCE node
I20260810 14:40:21.643635 16296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:21.644304 16296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:21.645476 16296 hybrid_clock.cc:648] HybridClock initialized: now 1786372821645450 us; error 40 us; skew 500 ppm
Aug 10 14:40:21 dist-test-slave-rmhg krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372821, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:21.648617 16296 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:40:21.649720 16296 webserver.cc:460] Webserver started at http://127.15.84.3:38927/ using document root <none> and password file <none>
I20260810 14:40:21.650333 16296 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:21.650400 16296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:21.650607 16296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:21.652948 16296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "c5dfbff1caaa42aeb5b47afa1162780d"
format_stamp: "Formatted at 2026-08-10 14:40:21 on dist-test-slave-rmhg"
server_key: "3ffc5f5e9a825834ba82bb5ac83fb5e2"
server_key_iv: "f7742310b064be47b5c421055fbd7b88"
server_key_version: "encryptionkey@0"
I20260810 14:40:21.653667 16296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "c5dfbff1caaa42aeb5b47afa1162780d"
format_stamp: "Formatted at 2026-08-10 14:40:21 on dist-test-slave-rmhg"
server_key: "3ffc5f5e9a825834ba82bb5ac83fb5e2"
server_key_iv: "f7742310b064be47b5c421055fbd7b88"
server_key_version: "encryptionkey@0"
I20260810 14:40:21.658041 16296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:40:21.660694 16322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:21.662176 16296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:21.662302 16296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "c5dfbff1caaa42aeb5b47afa1162780d"
format_stamp: "Formatted at 2026-08-10 14:40:21 on dist-test-slave-rmhg"
server_key: "3ffc5f5e9a825834ba82bb5ac83fb5e2"
server_key_iv: "f7742310b064be47b5c421055fbd7b88"
server_key_version: "encryptionkey@0"
I20260810 14:40:21.662413 16296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:21.675182 16296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:21.678491 16296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:21.678690 16296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:21.679297 16296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:21.680228 16296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:21.680295 16296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:21.680349 16296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:21.680374 16296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:21.689642 16296 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:35171
I20260810 14:40:21.689678 16497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:35171 every 8 connection(s)
I20260810 14:40:21.690659 16296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:40:21.697399 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 16296
I20260810 14:40:21.697535 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:40:21.697826 15696 external_mini_cluster.cc:1428] Setting key 15d67574b0a8721e90a89170e2159fc8
Aug 10 14:40:21 dist-test-slave-rmhg krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372821, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:21.706166 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:21.692781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:51441 (local address 127.15.84.62:35913)
0810 14:40:21.692915 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:21.692918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:21.693950 (+ 1032us) server_negotiation.cc:407] Connection header received
0810 14:40:21.695394 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:21.695397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:21.695463 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:21.695541 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:21.696928 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.697693 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.699277 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.699617 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.699713 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:21.702511 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:21.702515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:21.702517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:21.702547 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:21.703799 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:21.704335 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:21.704338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:21.704339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:21.704428 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:21.704742 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:21.704744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:21.704745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:21.704940 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:21.705032 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:21.705840 (+ 808us) server_negotiation.cc:299] Negotiation successful
0810 14:40:21.705991 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:40:21.707316 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:21.693208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:35913 (local address 127.15.84.3:51441)
0810 14:40:21.693761 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:21.693820 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:21.695138 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:21.695763 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:21.695770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:21.696202 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:21.696760 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:21.696777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.697866 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.697870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:21.699145 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:21.699158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.699793 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.699797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:21.700026 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:21.700711 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:21.700730 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:21.702307 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:21.703922 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:21.703929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:21.703945 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:21.704236 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:21.704513 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:21.704515 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:21.704517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:21.704673 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:21.705071 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:21.705078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:21.705532 (+ 454us) client_negotiation.cc:769] Sending connection context
0810 14:40:21.705795 (+ 263us) client_negotiation.cc:240] Negotiation successful
0810 14:40:21.706366 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":138,"threads_started":1}
I20260810 14:40:21.708695 16498 heartbeater.cc:347] Connected to a master server at 127.15.84.62:35913
I20260810 14:40:21.708988 16498 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:21.709607 16498 heartbeater.cc:511] Master 127.15.84.62:35913 requested a full tablet report, sending...
I20260810 14:40:21.710879 15776 ts_manager.cc:194] Registered new tserver with Master: c5dfbff1caaa42aeb5b47afa1162780d (127.15.84.3:35171)
I20260810 14:40:21.711651 15776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:51441
I20260810 14:40:21.712884 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 14:40:21 dist-test-slave-rmhg krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:40:21 dist-test-slave-rmhg krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.84.1@KRBTEST.COM
I20260810 14:40:21.742959 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:21.732586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51322 (local address 127.15.84.1:38025)
0810 14:40:21.732955 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:21.732971 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:21.733002 (+ 31us) server_negotiation.cc:407] Connection header received
0810 14:40:21.733282 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:21.733292 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:21.733520 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:21.733667 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:21.734399 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.735227 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.736075 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.736527 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.736628 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:21.739276 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:21.739291 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:21.739307 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:21.739339 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:21.740888 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:21.741452 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:21.741459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:21.741460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:21.741555 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:21.741837 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:21.741841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:21.741842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:21.742131 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:21.742370 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:21.742598 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 14:40:21.742774 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":133,"threads_started":1}
W20260810 14:40:21.744022 15939 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51322
Aug 10 14:40:21 dist-test-slave-rmhg krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:40:21 dist-test-slave-rmhg krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.1@KRBTEST.COM
I20260810 14:40:21.772207 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:21.763204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.15.84.1:38025)
0810 14:40:21.763387 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:21.763392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:21.763539 (+ 147us) server_negotiation.cc:407] Connection header received
0810 14:40:21.763703 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:21.763705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:21.763753 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:21.763819 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:21.764618 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.765224 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.766083 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:21.766349 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:21.766476 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:21.768952 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:21.768956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:21.768958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:21.768991 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:21.770415 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:21.770892 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:21.770896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:21.770898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:21.770991 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:21.771280 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:21.771284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:21.771285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:21.771489 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:21.771616 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:21.771885 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 14:40:21.772032 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 14:40:21.773479 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 15868
I20260810 14:40:21.791396 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 16078
I20260810 14:40:21.811932 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 16296
I20260810 14:40:21.857841 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 15726
2026-08-10T14:40:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2539 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:21 dist-test-slave-rmhg krb5kdc[16519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:21 dist-test-slave-rmhg krb5kdc[16519](info): set up 2 sockets
Aug 10 14:40:21 dist-test-slave-rmhg krb5kdc[16519](info): commencing operation
krb5kdc: starting...
W20260810 14:40:23.023768 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.001s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:40:23 dist-test-slave-rmhg krb5kdc[16519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:40:23.155828 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:37761
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:37761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:23.275074 16540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:23.275365 16540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:23.275408 16540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:23.279543 16540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:23.279649 16540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:23.279670 16540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:23.279685 16540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:23.279699 16540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:23.285305 16540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:37761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:37761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:23.286396 16540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:23.287423 16540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:23.296298 16547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:23.296304 16549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:23.296305 16546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:23.296610 16540 server_base.cc:1034] running on GCE node
I20260810 14:40:23.297452 16540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:23.298789 16540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:23.299971 16540 hybrid_clock.cc:648] HybridClock initialized: now 1786372823299949 us; error 51 us; skew 500 ppm
Aug 10 14:40:23 dist-test-slave-rmhg krb5kdc[16519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372823, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:23.303006 16540 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:40:23.304044 16540 webserver.cc:460] Webserver started at http://127.15.84.62:44897/ using document root <none> and password file <none>
I20260810 14:40:23.304656 16540 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:23.304731 16540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:23.304957 16540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:23.307171 16540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "02e50a6413ef485e84ce6091ab991d52"
format_stamp: "Formatted at 2026-08-10 14:40:23 on dist-test-slave-rmhg"
server_key: "951847d24ff7d88f82774414ad654c3e"
server_key_iv: "334bd4044a2482a7eec2946c88f7aca1"
server_key_version: "encryptionkey@0"
I20260810 14:40:23.307727 16540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "02e50a6413ef485e84ce6091ab991d52"
format_stamp: "Formatted at 2026-08-10 14:40:23 on dist-test-slave-rmhg"
server_key: "951847d24ff7d88f82774414ad654c3e"
server_key_iv: "334bd4044a2482a7eec2946c88f7aca1"
server_key_version: "encryptionkey@0"
I20260810 14:40:23.311720 16540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:40:23.314510 16563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:23.315829 16540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:40:23.315961 16540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "02e50a6413ef485e84ce6091ab991d52"
format_stamp: "Formatted at 2026-08-10 14:40:23 on dist-test-slave-rmhg"
server_key: "951847d24ff7d88f82774414ad654c3e"
server_key_iv: "334bd4044a2482a7eec2946c88f7aca1"
server_key_version: "encryptionkey@0"
I20260810 14:40:23.316074 16540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:23.347280 16540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:23.350692 16540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:23.350895 16540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:23.358265 16540 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:37761
I20260810 14:40:23.358299 16646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:37761 every 8 connection(s)
I20260810 14:40:23.359350 16540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:23.362361 16647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:23.363874 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 16540
I20260810 14:40:23.364005 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:23.364302 15696 external_mini_cluster.cc:1428] Setting key bf326df865ddf2a5a85d6e3e874f6614
I20260810 14:40:23.368989 16647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52: Bootstrap starting.
I20260810 14:40:23.371367 16647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:23.372170 16647 log.cc:824] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52: Log is configured to *not* fsync() on all Append() calls
Aug 10 14:40:23 dist-test-slave-rmhg krb5kdc[16519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:23.374778 16647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52: No bootstrap required, opened a new log
I20260810 14:40:23.379863 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:23.365955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47716 (local address 127.15.84.62:37761)
0810 14:40:23.366455 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:23.366471 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:23.366510 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:40:23.367288 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:23.367312 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:23.367680 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:23.368021 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:23.368834 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:23.369934 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:23.370806 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:23.371322 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:23.371482 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:23.374381 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:23.374387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:23.374402 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:23.374432 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:23.376463 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:23.376946 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:23.376953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:23.376960 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:23.377126 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:23.377441 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:23.377445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:23.377447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:23.378035 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:23.378253 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:23.378448 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 14:40:23.378795 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":162,"threads_started":1}
I20260810 14:40:23.379674 16647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e50a6413ef485e84ce6091ab991d52" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37761 } }
I20260810 14:40:23.379967 16647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:23.380012 16647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e50a6413ef485e84ce6091ab991d52, State: Initialized, Role: FOLLOWER
I20260810 14:40:23.380530 16647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e50a6413ef485e84ce6091ab991d52" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37761 } }
I20260810 14:40:23.380712 16647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:23.380784 16647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:23.380887 16647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:23.381948 16647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e50a6413ef485e84ce6091ab991d52" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37761 } }
I20260810 14:40:23.382332 16647 leader_election.cc:302] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02e50a6413ef485e84ce6091ab991d52; no voters:
I20260810 14:40:23.382730 16647 leader_election.cc:288] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:23.382853 16655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:23.383098 16655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [term 1 LEADER]: Becoming Leader. State: Replica: 02e50a6413ef485e84ce6091ab991d52, State: Running, Role: LEADER
I20260810 14:40:23.383416 16655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e50a6413ef485e84ce6091ab991d52" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37761 } }
I20260810 14:40:23.384054 16647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:23.385138 16657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02e50a6413ef485e84ce6091ab991d52. Latest consensus state: current_term: 1 leader_uuid: "02e50a6413ef485e84ce6091ab991d52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e50a6413ef485e84ce6091ab991d52" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37761 } } }
I20260810 14:40:23.385306 16657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:23.385602 16656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02e50a6413ef485e84ce6091ab991d52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02e50a6413ef485e84ce6091ab991d52" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37761 } } }
I20260810 14:40:23.385706 16656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:23.385810 16667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:23.388870 16667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:23.394280 16667 catalog_manager.cc:1347] Generated new cluster ID: 557a998fa8554f5f9706768699f7ecd7
I20260810 14:40:23.394361 16667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:23.407582 16667 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:23.408567 16667 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:23.414806 16667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02e50a6413ef485e84ce6091ab991d52: Generated new TSK 0
I20260810 14:40:23.415611 16667 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:40:23.451939 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:37761
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:23.572372 16689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:23.572693 16689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:23.572743 16689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:23.576994 16689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:23.577096 16689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:23.577196 16689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:40:23.582902 16689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:37761
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:23.584103 16689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:23.585292 16689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:40:23.595458 16689 server_base.cc:1034] running on GCE node
W20260810 14:40:23.595402 16700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:23.595402 16695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:23.595429 16696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:23.596143 16689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:23.596834 16689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:23.598001 16689 hybrid_clock.cc:648] HybridClock initialized: now 1786372823597952 us; error 62 us; skew 500 ppm
Aug 10 14:40:23 dist-test-slave-rmhg krb5kdc[16519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372823, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:23.600878 16689 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:40:23.601900 16689 webserver.cc:460] Webserver started at http://127.15.84.1:35537/ using document root <none> and password file <none>
I20260810 14:40:23.602625 16689 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:23.602731 16689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:23.602994 16689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:23.605315 16689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "77c3adee6ab74528a52ad59fa421427f"
format_stamp: "Formatted at 2026-08-10 14:40:23 on dist-test-slave-rmhg"
server_key: "83af9cc47b13408d4c239e42440704cd"
server_key_iv: "69fe44477607c6189025fa3bc21000a8"
server_key_version: "encryptionkey@0"
I20260810 14:40:23.605901 16689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "77c3adee6ab74528a52ad59fa421427f"
format_stamp: "Formatted at 2026-08-10 14:40:23 on dist-test-slave-rmhg"
server_key: "83af9cc47b13408d4c239e42440704cd"
server_key_iv: "69fe44477607c6189025fa3bc21000a8"
server_key_version: "encryptionkey@0"
I20260810 14:40:23.610248 16689 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:40:23.613235 16709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:23.614696 16689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:23.614832 16689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "77c3adee6ab74528a52ad59fa421427f"
format_stamp: "Formatted at 2026-08-10 14:40:23 on dist-test-slave-rmhg"
server_key: "83af9cc47b13408d4c239e42440704cd"
server_key_iv: "69fe44477607c6189025fa3bc21000a8"
server_key_version: "encryptionkey@0"
I20260810 14:40:23.614956 16689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:23.632298 16689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:23.635486 16689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:23.635667 16689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:23.636256 16689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:23.637115 16689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:23.637193 16689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:23.637249 16689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:23.637272 16689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:23.646512 16689 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:41725
I20260810 14:40:23.646621 16884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:41725 every 8 connection(s)
I20260810 14:40:23.647559 16689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:40:23.650068 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 16689
I20260810 14:40:23.650189 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:40:23.650508 15696 external_mini_cluster.cc:1428] Setting key a985b6ee51396aa76609b4686e2d2ee7
Aug 10 14:40:23 dist-test-slave-rmhg krb5kdc[16519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372823, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:40:23.663720 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:23.649593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:48143 (local address 127.15.84.62:37761)
0810 14:40:23.649783 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:23.649787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:23.650777 (+ 990us) server_negotiation.cc:407] Connection header received
0810 14:40:23.652283 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:23.652287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:23.652345 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:23.652420 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:23.653909 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:23.654467 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:23.655915 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:23.656183 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:23.656280 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:23.659716 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:23.659720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:23.659723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:23.659754 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:23.661147 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:23.661851 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:23.661855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:23.661856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:23.662029 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:23.662456 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:23.662460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:23.662461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:23.662741 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:23.662829 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:23.663430 (+ 601us) server_negotiation.cc:299] Negotiation successful
0810 14:40:23.663587 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
I20260810 14:40:23.664775 16891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:23.650048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37761 (local address 127.15.84.1:48143)
0810 14:40:23.650613 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:23.650673 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:23.652059 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:23.652665 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:23.652674 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:23.653105 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:23.653740 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:23.653757 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:23.654613 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:23.654616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:23.655750 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:23.655764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:23.656371 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:23.656376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:23.656604 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:23.657489 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:23.657513 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:23.659459 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:23.661313 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:23.661321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:23.661345 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:23.661713 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:23.662165 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:23.662168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:23.662170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:23.662346 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:23.662840 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:23.662846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:23.663157 (+ 311us) client_negotiation.cc:769] Sending connection context
0810 14:40:23.663340 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 14:40:23.663679 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":148,"threads_started":1}
I20260810 14:40:23.666205 16887 heartbeater.cc:347] Connected to a master server at 127.15.84.62:37761
I20260810 14:40:23.666482 16887 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:23.667155 16887 heartbeater.cc:511] Master 127.15.84.62:37761 requested a full tablet report, sending...
I20260810 14:40:23.668865 16589 ts_manager.cc:194] Registered new tserver with Master: 77c3adee6ab74528a52ad59fa421427f (127.15.84.1:41725)
I20260810 14:40:23.670146 16589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:48143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:40:23.684546 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:37761
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:23.803795 16895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:23.804096 16895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:23.804144 16895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:23.808283 16895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:23.808365 16895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:23.808442 16895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:40:23.814143 16895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:37761
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:23.815349 16895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:23.816498 16895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:23.826917 16908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:23.827001 16895 server_base.cc:1034] running on GCE node
W20260810 14:40:23.826911 16903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:23.826911 16910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:23.827693 16895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:23.828357 16895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:23.829512 16895 hybrid_clock.cc:648] HybridClock initialized: now 1786372823829487 us; error 40 us; skew 500 ppm
Aug 10 14:40:23 dist-test-slave-rmhg krb5kdc[16519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372823, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:23.832482 16895 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:40:23.833547 16895 webserver.cc:460] Webserver started at http://127.15.84.2:35217/ using document root <none> and password file <none>
I20260810 14:40:23.834139 16895 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:23.834205 16895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:23.834388 16895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:23.836558 16895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "bfab4ee4bdc6415bb8ea538b0da51024"
format_stamp: "Formatted at 2026-08-10 14:40:23 on dist-test-slave-rmhg"
server_key: "13d4e88014d5d82218c061c577c4f84f"
server_key_iv: "9ae08e8b44ab42c1163655a8c2183fe5"
server_key_version: "encryptionkey@0"
I20260810 14:40:23.837111 16895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "bfab4ee4bdc6415bb8ea538b0da51024"
format_stamp: "Formatted at 2026-08-10 14:40:23 on dist-test-slave-rmhg"
server_key: "13d4e88014d5d82218c061c577c4f84f"
server_key_iv: "9ae08e8b44ab42c1163655a8c2183fe5"
server_key_version: "encryptionkey@0"
I20260810 14:40:23.841068 16895 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 14:40:23.843775 16925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:23.845088 16895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:40:23.845259 16895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "bfab4ee4bdc6415bb8ea538b0da51024"
format_stamp: "Formatted at 2026-08-10 14:40:23 on dist-test-slave-rmhg"
server_key: "13d4e88014d5d82218c061c577c4f84f"
server_key_iv: "9ae08e8b44ab42c1163655a8c2183fe5"
server_key_version: "encryptionkey@0"
I20260810 14:40:23.845373 16895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:23.861430 16895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:23.864409 16895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:23.864604 16895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:23.865252 16895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:23.866159 16895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:23.866211 16895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:23.866252 16895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:23.866272 16895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:23.875306 16895 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:45127
I20260810 14:40:23.875331 17098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:45127 every 8 connection(s)
I20260810 14:40:23.876297 16895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:40:23.882401 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 16895
I20260810 14:40:23.882512 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:40:23.882784 15696 external_mini_cluster.cc:1428] Setting key 39fec2aa3efff20832ea4bef5deed265
Aug 10 14:40:23 dist-test-slave-rmhg krb5kdc[16519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372823, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:23.892704 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:23.878282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:42447 (local address 127.15.84.62:37761)
0810 14:40:23.878421 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:23.878425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:23.879451 (+ 1026us) server_negotiation.cc:407] Connection header received
0810 14:40:23.880924 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:23.880929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:23.881001 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:23.881108 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:23.882516 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:23.883141 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:23.884914 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:23.885226 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:23.885320 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:23.888482 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:23.888486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:23.888490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:23.888524 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:23.890307 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:23.890888 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:23.890892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:23.890893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:23.890997 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:23.891386 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:23.891390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:23.891392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:23.891656 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:23.891767 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:23.892357 (+ 590us) server_negotiation.cc:299] Negotiation successful
0810 14:40:23.892518 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:40:23.893683 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:23.878725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37761 (local address 127.15.84.2:42447)
0810 14:40:23.879285 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:23.879333 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:23.880657 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:23.881340 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:23.881351 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:23.881777 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:23.882341 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:23.882358 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:23.883382 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:23.883386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:23.884769 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:23.884782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:23.885397 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:23.885402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:23.885645 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:23.886330 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:23.886353 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:23.888253 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:23.890457 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:23.890463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:23.890482 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:23.890773 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:23.891118 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:23.891121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:23.891123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:23.891288 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:23.891773 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:23.891779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:23.892091 (+ 312us) client_negotiation.cc:769] Sending connection context
0810 14:40:23.892283 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 14:40:23.892604 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":173,"threads_started":1}
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
I20260810 14:40:23.894970 17099 heartbeater.cc:347] Connected to a master server at 127.15.84.62:37761
I20260810 14:40:23.895231 17099 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:23.895782 17099 heartbeater.cc:511] Master 127.15.84.62:37761 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
I20260810 14:40:23.896948 16589 ts_manager.cc:194] Registered new tserver with Master: bfab4ee4bdc6415bb8ea538b0da51024 (127.15.84.2:45127)
I20260810 14:40:23.897629 16589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:42447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:40:23.911983 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:37761
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:24.032321 17108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:24.032629 17108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:24.032675 17108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:24.036742 17108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:24.036816 17108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:24.036877 17108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:40:24.042459 17108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:37761
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:24.043628 17108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:24.044715 17108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:24.055493 17118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:24.055513 17115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:24.055615 17108 server_base.cc:1034] running on GCE node
W20260810 14:40:24.055493 17116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:24.056326 17108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:24.057065 17108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:24.058264 17108 hybrid_clock.cc:648] HybridClock initialized: now 1786372824058223 us; error 56 us; skew 500 ppm
Aug 10 14:40:24 dist-test-slave-rmhg krb5kdc[16519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372824, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:24.061338 17108 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:40:24.062325 17108 webserver.cc:460] Webserver started at http://127.15.84.3:35431/ using document root <none> and password file <none>
I20260810 14:40:24.062891 17108 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:24.062953 17108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:24.063138 17108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:24.065382 17108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "06d8aae099054e589f074007b8f10ad5"
format_stamp: "Formatted at 2026-08-10 14:40:24 on dist-test-slave-rmhg"
server_key: "709564975005d53bac546dd4587a5e8a"
server_key_iv: "0f6743dade191d60fb9326e405f06ee0"
server_key_version: "encryptionkey@0"
I20260810 14:40:24.065961 17108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "06d8aae099054e589f074007b8f10ad5"
format_stamp: "Formatted at 2026-08-10 14:40:24 on dist-test-slave-rmhg"
server_key: "709564975005d53bac546dd4587a5e8a"
server_key_iv: "0f6743dade191d60fb9326e405f06ee0"
server_key_version: "encryptionkey@0"
I20260810 14:40:24.070163 17108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:40:24.072961 17127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:24.074340 17108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:24.074468 17108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "06d8aae099054e589f074007b8f10ad5"
format_stamp: "Formatted at 2026-08-10 14:40:24 on dist-test-slave-rmhg"
server_key: "709564975005d53bac546dd4587a5e8a"
server_key_iv: "0f6743dade191d60fb9326e405f06ee0"
server_key_version: "encryptionkey@0"
I20260810 14:40:24.074584 17108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:24.103363 17108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:24.106994 17108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:24.107201 17108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:24.107856 17108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:24.108772 17108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:24.108825 17108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:24.108882 17108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:24.108907 17108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:24.117982 17108 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:34321
I20260810 14:40:24.118014 17326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:34321 every 8 connection(s)
I20260810 14:40:24.119036 17108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:40:24.119877 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 17108
I20260810 14:40:24.120010 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:40:24.120286 15696 external_mini_cluster.cc:1428] Setting key 5abf4ebd7a2fff11867e47fe725074a0
Aug 10 14:40:24 dist-test-slave-rmhg krb5kdc[16519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372824, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:24.133798 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.122080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:52231 (local address 127.15.84.62:37761)
0810 14:40:24.122228 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:24.122233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:24.123268 (+ 1035us) server_negotiation.cc:407] Connection header received
0810 14:40:24.124566 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:24.124569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:24.124621 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:24.124692 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:24.125935 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.126469 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.127636 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.127857 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.127929 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.130387 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:24.130391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:24.130394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:24.130421 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:24.131626 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:24.132135 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:24.132138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:24.132139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:24.132218 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:24.132496 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:24.132502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:24.132503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:24.132722 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:24.132798 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:24.133469 (+ 671us) server_negotiation.cc:299] Negotiation successful
0810 14:40:24.133643 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:40:24.134961 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.122621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37761 (local address 127.15.84.3:52231)
0810 14:40:24.123112 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:24.123166 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:24.124354 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:24.124855 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:24.124862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:24.125281 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:24.125815 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:24.125830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.126577 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.126579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:24.127513 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:24.127522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.127971 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.127973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:24.128124 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.128670 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:24.128688 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:24.130206 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:24.131751 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:24.131757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:24.131773 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:24.132052 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:24.132283 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:24.132285 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:24.132287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:24.132430 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:24.132819 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:24.132827 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:24.133225 (+ 398us) client_negotiation.cc:769] Sending connection context
0810 14:40:24.133441 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 14:40:24.133869 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":137,"threads_started":1}
I20260810 14:40:24.136324 17329 heartbeater.cc:347] Connected to a master server at 127.15.84.62:37761
I20260810 14:40:24.136621 17329 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:24.137292 17329 heartbeater.cc:511] Master 127.15.84.62:37761 requested a full tablet report, sending...
I20260810 14:40:24.138600 16589 ts_manager.cc:194] Registered new tserver with Master: 06d8aae099054e589f074007b8f10ad5 (127.15.84.3:34321)
I20260810 14:40:24.139197 16589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:52231
I20260810 14:40:24.146816 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:40:24.157560 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.150695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47732 (local address 127.15.84.62:37761)
0810 14:40:24.150832 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:24.150836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:24.150987 (+ 151us) server_negotiation.cc:407] Connection header received
0810 14:40:24.151127 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:24.151131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:24.151176 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:24.151239 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:24.151930 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.152459 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.153344 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.153567 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.153674 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.154902 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:24.154905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:24.154907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:24.154933 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:24.156056 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:24.156419 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:24.156421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:24.156423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:24.156489 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:24.156777 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:24.156779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:24.156781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:24.156994 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:24.157098 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:24.157334 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 14:40:24.157437 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:40:24.162303 16589 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:40:24.165450 16589 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:24.180419 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.174888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:45127 (local address 127.0.0.1:34316)
0810 14:40:24.175184 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:24.175198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:24.175284 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:24.176416 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:24.176419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:24.176438 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:24.176523 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:24.176529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.177916 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.177919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:24.179006 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:24.179013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.179857 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.179861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:24.179968 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.180006 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 14:40:24.180122 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 14:40:24.180253 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260810 14:40:24.180454 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.175598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34316 (local address 127.15.84.2:45127)
0810 14:40:24.176000 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:24.176013 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:24.176035 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:40:24.176105 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:24.176109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:24.176285 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:24.176429 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:24.176657 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.177743 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.179112 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.179746 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.179844 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.180008 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:24.180174 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 14:40:24.180285 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":141,"threads_started":1}
I20260810 14:40:24.180805 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.175080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.15.84.1:41725)
0810 14:40:24.175494 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:24.175501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:24.175524 (+ 23us) server_negotiation.cc:407] Connection header received
0810 14:40:24.175615 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:24.175620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:24.175824 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:24.175960 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:24.176299 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.177143 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.178952 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.179775 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.179881 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.180022 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:24.180502 (+ 480us) server_negotiation.cc:299] Negotiation successful
0810 14:40:24.180622 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":112,"threads_started":1}
I20260810 14:40:24.180889 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.174314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:41725 (local address 127.0.0.1:56488)
0810 14:40:24.174729 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:24.174745 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:24.174843 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:24.176083 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:24.176085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:24.176100 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:24.176187 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:24.176191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.177313 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.177317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:24.178775 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:24.178787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.180090 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.180093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:24.180239 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.180273 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 14:40:24.180340 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 14:40:24.180417 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":102,"thread_start_us":125,"threads_started":1}
I20260810 14:40:24.181354 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.174321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46926 (local address 127.15.84.3:34321)
0810 14:40:24.175060 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:24.175073 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:24.175097 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:40:24.175177 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:24.175182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:24.175405 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:24.175565 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:24.175934 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.177108 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.178901 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.179775 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.179881 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.180022 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:24.181086 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0810 14:40:24.181233 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":138,"threads_started":1}
I20260810 14:40:24.181596 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.174118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:34321 (local address 127.0.0.1:46926)
0810 14:40:24.174479 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:24.174530 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:24.174661 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:24.175622 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:24.175628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:24.175652 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:24.175788 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:24.175796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.177265 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.177268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:24.178681 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:24.178690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.180085 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.180088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:24.180276 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.180298 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:40:24.180364 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 14:40:24.180437 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":128,"thread_start_us":95,"threads_started":1}
I20260810 14:40:24.183593 16986 tablet_service.cc:1467] Processing CreateTablet for tablet 9820f8d125cc41f7ae4d57bcd970cd8b (DEFAULT_TABLE table=test-table [id=fa83705defff4e5bad04dcf828c5ff5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:24.183790 16791 tablet_service.cc:1467] Processing CreateTablet for tablet 9820f8d125cc41f7ae4d57bcd970cd8b (DEFAULT_TABLE table=test-table [id=fa83705defff4e5bad04dcf828c5ff5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:24.184227 17204 tablet_service.cc:1467] Processing CreateTablet for tablet 9820f8d125cc41f7ae4d57bcd970cd8b (DEFAULT_TABLE table=test-table [id=fa83705defff4e5bad04dcf828c5ff5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:24.184837 16986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9820f8d125cc41f7ae4d57bcd970cd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:24.184926 16791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9820f8d125cc41f7ae4d57bcd970cd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:24.185271 17204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9820f8d125cc41f7ae4d57bcd970cd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:24.191471 17356 tablet_bootstrap.cc:492] T 9820f8d125cc41f7ae4d57bcd970cd8b P bfab4ee4bdc6415bb8ea538b0da51024: Bootstrap starting.
I20260810 14:40:24.193094 17357 tablet_bootstrap.cc:492] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f: Bootstrap starting.
I20260810 14:40:24.193516 17358 tablet_bootstrap.cc:492] T 9820f8d125cc41f7ae4d57bcd970cd8b P 06d8aae099054e589f074007b8f10ad5: Bootstrap starting.
I20260810 14:40:24.194630 17356 tablet_bootstrap.cc:654] T 9820f8d125cc41f7ae4d57bcd970cd8b P bfab4ee4bdc6415bb8ea538b0da51024: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:24.195511 17358 tablet_bootstrap.cc:654] T 9820f8d125cc41f7ae4d57bcd970cd8b P 06d8aae099054e589f074007b8f10ad5: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:24.195511 17357 tablet_bootstrap.cc:654] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:24.195808 17356 log.cc:824] T 9820f8d125cc41f7ae4d57bcd970cd8b P bfab4ee4bdc6415bb8ea538b0da51024: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:24.196422 17357 log.cc:824] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:24.196771 17358 log.cc:824] T 9820f8d125cc41f7ae4d57bcd970cd8b P 06d8aae099054e589f074007b8f10ad5: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:24.197988 17356 tablet_bootstrap.cc:492] T 9820f8d125cc41f7ae4d57bcd970cd8b P bfab4ee4bdc6415bb8ea538b0da51024: No bootstrap required, opened a new log
I20260810 14:40:24.198206 17356 ts_tablet_manager.cc:1397] T 9820f8d125cc41f7ae4d57bcd970cd8b P bfab4ee4bdc6415bb8ea538b0da51024: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 14:40:24.198634 17357 tablet_bootstrap.cc:492] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f: No bootstrap required, opened a new log
I20260810 14:40:24.198652 17358 tablet_bootstrap.cc:492] T 9820f8d125cc41f7ae4d57bcd970cd8b P 06d8aae099054e589f074007b8f10ad5: No bootstrap required, opened a new log
I20260810 14:40:24.198839 17358 ts_tablet_manager.cc:1397] T 9820f8d125cc41f7ae4d57bcd970cd8b P 06d8aae099054e589f074007b8f10ad5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 14:40:24.198839 17357 ts_tablet_manager.cc:1397] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:40:24.201946 17356 raft_consensus.cc:348] T 9820f8d125cc41f7ae4d57bcd970cd8b P bfab4ee4bdc6415bb8ea538b0da51024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d8aae099054e589f074007b8f10ad5" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34321 } } peers { permanent_uuid: "bfab4ee4bdc6415bb8ea538b0da51024" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45127 } } peers { permanent_uuid: "77c3adee6ab74528a52ad59fa421427f" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41725 } }
I20260810 14:40:24.202234 17356 raft_consensus.cc:374] T 9820f8d125cc41f7ae4d57bcd970cd8b P bfab4ee4bdc6415bb8ea538b0da51024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:24.202292 17356 raft_consensus.cc:729] T 9820f8d125cc41f7ae4d57bcd970cd8b P bfab4ee4bdc6415bb8ea538b0da51024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfab4ee4bdc6415bb8ea538b0da51024, State: Initialized, Role: FOLLOWER
I20260810 14:40:24.202766 17356 consensus_queue.cc:260] T 9820f8d125cc41f7ae4d57bcd970cd8b P bfab4ee4bdc6415bb8ea538b0da51024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d8aae099054e589f074007b8f10ad5" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34321 } } peers { permanent_uuid: "bfab4ee4bdc6415bb8ea538b0da51024" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45127 } } peers { permanent_uuid: "77c3adee6ab74528a52ad59fa421427f" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41725 } }
I20260810 14:40:24.202883 17358 raft_consensus.cc:348] T 9820f8d125cc41f7ae4d57bcd970cd8b P 06d8aae099054e589f074007b8f10ad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d8aae099054e589f074007b8f10ad5" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34321 } } peers { permanent_uuid: "bfab4ee4bdc6415bb8ea538b0da51024" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45127 } } peers { permanent_uuid: "77c3adee6ab74528a52ad59fa421427f" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41725 } }
I20260810 14:40:24.203194 17358 raft_consensus.cc:374] T 9820f8d125cc41f7ae4d57bcd970cd8b P 06d8aae099054e589f074007b8f10ad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:24.203255 17358 raft_consensus.cc:729] T 9820f8d125cc41f7ae4d57bcd970cd8b P 06d8aae099054e589f074007b8f10ad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d8aae099054e589f074007b8f10ad5, State: Initialized, Role: FOLLOWER
I20260810 14:40:24.203332 17357 raft_consensus.cc:348] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d8aae099054e589f074007b8f10ad5" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34321 } } peers { permanent_uuid: "bfab4ee4bdc6415bb8ea538b0da51024" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45127 } } peers { permanent_uuid: "77c3adee6ab74528a52ad59fa421427f" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41725 } }
I20260810 14:40:24.203593 17357 raft_consensus.cc:374] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:24.203619 17099 heartbeater.cc:503] Master 127.15.84.62:37761 was elected leader, sending a full tablet report...
I20260810 14:40:24.203650 17357 raft_consensus.cc:729] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c3adee6ab74528a52ad59fa421427f, State: Initialized, Role: FOLLOWER
I20260810 14:40:24.203814 17358 consensus_queue.cc:260] T 9820f8d125cc41f7ae4d57bcd970cd8b P 06d8aae099054e589f074007b8f10ad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d8aae099054e589f074007b8f10ad5" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34321 } } peers { permanent_uuid: "bfab4ee4bdc6415bb8ea538b0da51024" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45127 } } peers { permanent_uuid: "77c3adee6ab74528a52ad59fa421427f" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41725 } }
I20260810 14:40:24.204080 17357 consensus_queue.cc:260] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d8aae099054e589f074007b8f10ad5" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34321 } } peers { permanent_uuid: "bfab4ee4bdc6415bb8ea538b0da51024" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45127 } } peers { permanent_uuid: "77c3adee6ab74528a52ad59fa421427f" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41725 } }
I20260810 14:40:24.204649 17329 heartbeater.cc:503] Master 127.15.84.62:37761 was elected leader, sending a full tablet report...
I20260810 14:40:24.205008 17358 ts_tablet_manager.cc:1428] T 9820f8d125cc41f7ae4d57bcd970cd8b P 06d8aae099054e589f074007b8f10ad5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 14:40:24.205008 17357 ts_tablet_manager.cc:1428] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 14:40:24.205353 16887 heartbeater.cc:503] Master 127.15.84.62:37761 was elected leader, sending a full tablet report...
I20260810 14:40:24.207089 17356 ts_tablet_manager.cc:1428] T 9820f8d125cc41f7ae4d57bcd970cd8b P bfab4ee4bdc6415bb8ea538b0da51024: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260810 14:40:24.361939 17369 raft_consensus.cc:482] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:24.362221 17369 raft_consensus.cc:504] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d8aae099054e589f074007b8f10ad5" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34321 } } peers { permanent_uuid: "bfab4ee4bdc6415bb8ea538b0da51024" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45127 } } peers { permanent_uuid: "77c3adee6ab74528a52ad59fa421427f" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41725 } }
I20260810 14:40:24.363679 17369 leader_election.cc:288] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06d8aae099054e589f074007b8f10ad5 (127.15.84.3:34321), bfab4ee4bdc6415bb8ea538b0da51024 (127.15.84.2:45127)
I20260810 14:40:24.368960 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.364656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:46473 (local address 127.15.84.2:45127)
0810 14:40:24.364852 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:24.364857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:24.365001 (+ 144us) server_negotiation.cc:407] Connection header received
0810 14:40:24.365136 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:24.365139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:24.365212 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:24.365291 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:24.365586 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.366310 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.367704 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.368244 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.368321 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.368380 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:24.368744 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 14:40:24.368849 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260810 14:40:24.369287 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.364102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:38521 (local address 127.15.84.3:34321)
0810 14:40:24.364262 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:24.364267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:24.364566 (+ 299us) server_negotiation.cc:407] Connection header received
0810 14:40:24.364849 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:24.364852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:24.364909 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:24.364950 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:24.365459 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.366270 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.367895 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.368548 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.368629 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.368695 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:24.369049 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 14:40:24.369173 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:40:24.369307 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.364001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:34321 (local address 127.15.84.1:38521)
0810 14:40:24.364453 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:24.364484 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:24.364701 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:24.365111 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:24.365114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:24.365134 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:24.365299 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:24.365308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.366404 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.366408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:24.367730 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:24.367742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.368797 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.368800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:24.368915 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.368936 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 14:40:24.369001 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 14:40:24.369076 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":47,"thread_start_us":152,"threads_started":1}
I20260810 14:40:24.369637 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.364514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:45127 (local address 127.15.84.1:46473)
0810 14:40:24.364893 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:24.364907 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:24.365025 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:24.365313 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:24.365316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:24.365335 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:24.365464 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:24.365470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.366443 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.366447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:24.367540 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:24.367552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.368469 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.368471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:24.368577 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.368612 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 14:40:24.369454 (+ 842us) client_negotiation.cc:240] Negotiation successful
0810 14:40:24.369524 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":3,"thread_start_us":102,"threads_started":1}
W20260810 14:40:24.370060 17330 tablet.cc:2369] T 9820f8d125cc41f7ae4d57bcd970cd8b P 06d8aae099054e589f074007b8f10ad5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:40:24.370108 17249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9820f8d125cc41f7ae4d57bcd970cd8b" candidate_uuid: "77c3adee6ab74528a52ad59fa421427f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d8aae099054e589f074007b8f10ad5" is_pre_election: true
I20260810 14:40:24.370340 17018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9820f8d125cc41f7ae4d57bcd970cd8b" candidate_uuid: "77c3adee6ab74528a52ad59fa421427f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfab4ee4bdc6415bb8ea538b0da51024" is_pre_election: true
I20260810 14:40:24.370450 17249 raft_consensus.cc:2454] T 9820f8d125cc41f7ae4d57bcd970cd8b P 06d8aae099054e589f074007b8f10ad5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c3adee6ab74528a52ad59fa421427f in term 0.
I20260810 14:40:24.370644 17018 raft_consensus.cc:2454] T 9820f8d125cc41f7ae4d57bcd970cd8b P bfab4ee4bdc6415bb8ea538b0da51024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c3adee6ab74528a52ad59fa421427f in term 0.
I20260810 14:40:24.371201 16712 leader_election.cc:302] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c3adee6ab74528a52ad59fa421427f, bfab4ee4bdc6415bb8ea538b0da51024; no voters:
I20260810 14:40:24.371517 17369 raft_consensus.cc:2781] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:40:24.371614 17369 raft_consensus.cc:482] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:40:24.371659 17369 raft_consensus.cc:3037] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:24.372783 17369 raft_consensus.cc:504] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d8aae099054e589f074007b8f10ad5" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34321 } } peers { permanent_uuid: "bfab4ee4bdc6415bb8ea538b0da51024" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45127 } } peers { permanent_uuid: "77c3adee6ab74528a52ad59fa421427f" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41725 } }
I20260810 14:40:24.373183 17369 leader_election.cc:288] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [CANDIDATE]: Term 1 election: Requested vote from peers 06d8aae099054e589f074007b8f10ad5 (127.15.84.3:34321), bfab4ee4bdc6415bb8ea538b0da51024 (127.15.84.2:45127)
I20260810 14:40:24.373620 17249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9820f8d125cc41f7ae4d57bcd970cd8b" candidate_uuid: "77c3adee6ab74528a52ad59fa421427f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d8aae099054e589f074007b8f10ad5"
I20260810 14:40:24.373652 17018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9820f8d125cc41f7ae4d57bcd970cd8b" candidate_uuid: "77c3adee6ab74528a52ad59fa421427f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfab4ee4bdc6415bb8ea538b0da51024"
I20260810 14:40:24.373780 17018 raft_consensus.cc:3037] T 9820f8d125cc41f7ae4d57bcd970cd8b P bfab4ee4bdc6415bb8ea538b0da51024 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:24.373759 17249 raft_consensus.cc:3037] T 9820f8d125cc41f7ae4d57bcd970cd8b P 06d8aae099054e589f074007b8f10ad5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:24.374886 17249 raft_consensus.cc:2454] T 9820f8d125cc41f7ae4d57bcd970cd8b P 06d8aae099054e589f074007b8f10ad5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c3adee6ab74528a52ad59fa421427f in term 1.
I20260810 14:40:24.374891 17018 raft_consensus.cc:2454] T 9820f8d125cc41f7ae4d57bcd970cd8b P bfab4ee4bdc6415bb8ea538b0da51024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c3adee6ab74528a52ad59fa421427f in term 1.
I20260810 14:40:24.375249 16712 leader_election.cc:302] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d8aae099054e589f074007b8f10ad5, 77c3adee6ab74528a52ad59fa421427f; no voters:
I20260810 14:40:24.375450 17369 raft_consensus.cc:2781] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:24.375715 17369 raft_consensus.cc:686] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [term 1 LEADER]: Becoming Leader. State: Replica: 77c3adee6ab74528a52ad59fa421427f, State: Running, Role: LEADER
I20260810 14:40:24.376142 17369 consensus_queue.cc:237] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d8aae099054e589f074007b8f10ad5" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34321 } } peers { permanent_uuid: "bfab4ee4bdc6415bb8ea538b0da51024" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45127 } } peers { permanent_uuid: "77c3adee6ab74528a52ad59fa421427f" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41725 } }
W20260810 14:40:24.377526 17101 tablet.cc:2369] T 9820f8d125cc41f7ae4d57bcd970cd8b P bfab4ee4bdc6415bb8ea538b0da51024: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:40:24.380368 16584 catalog_manager.cc:5495] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f reported cstate change: term changed from 0 to 1, leader changed from <none> to 77c3adee6ab74528a52ad59fa421427f (127.15.84.1). New cstate: current_term: 1 leader_uuid: "77c3adee6ab74528a52ad59fa421427f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d8aae099054e589f074007b8f10ad5" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfab4ee4bdc6415bb8ea538b0da51024" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77c3adee6ab74528a52ad59fa421427f" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41725 } health_report { overall_health: HEALTHY } } }
W20260810 14:40:24.399096 16889 tablet.cc:2369] T 9820f8d125cc41f7ae4d57bcd970cd8b P 77c3adee6ab74528a52ad59fa421427f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 14:40:24 dist-test-slave-rmhg krb5kdc[16519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.1@KRBTEST.COM
I20260810 14:40:24.430224 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.415324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.15.84.1:41725)
0810 14:40:24.415490 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:24.415495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:24.415588 (+ 93us) server_negotiation.cc:407] Connection header received
0810 14:40:24.415731 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:24.415734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:24.415784 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:24.415852 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:24.416591 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.417131 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.418093 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.418340 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.418438 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.426656 (+ 8218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:24.426675 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:24.426692 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:24.426732 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:24.428231 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:24.428694 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:24.428700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:24.428701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:24.428795 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:24.429069 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:24.429073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:24.429075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:24.429439 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:24.429686 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:24.429901 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 14:40:24.430037 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 14:40:24 dist-test-slave-rmhg krb5kdc[16519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:40:24 dist-test-slave-rmhg krb5kdc[16519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.84.1@KRBTEST.COM
I20260810 14:40:24.459112 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.450422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.15.84.1:41725)
0810 14:40:24.450577 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:24.450582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:24.450699 (+ 117us) server_negotiation.cc:407] Connection header received
0810 14:40:24.450863 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:24.450866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:24.450911 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:24.450979 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:24.451721 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.452257 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.453122 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.453440 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.453539 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.456105 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:24.456109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:24.456111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:24.456151 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:24.457409 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:24.457847 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:24.457852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:24.457853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:24.457938 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:24.458199 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:24.458202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:24.458204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:24.458393 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:24.458540 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:24.458817 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 14:40:24.458947 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 14:40:24 dist-test-slave-rmhg krb5kdc[16519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:40:24 dist-test-slave-rmhg krb5kdc[16519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.1@KRBTEST.COM
I20260810 14:40:24.488214 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:24.479381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.15.84.1:41725)
0810 14:40:24.479602 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:24.479607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:24.479718 (+ 111us) server_negotiation.cc:407] Connection header received
0810 14:40:24.479875 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:24.479878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:24.479931 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:24.480003 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:24.480747 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.481340 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.482229 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:24.482453 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:24.482553 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:24.485118 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:24.485121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:24.485123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:24.485197 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:24.486568 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:24.487003 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:24.487006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:24.487008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:24.487091 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:24.487350 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:24.487353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:24.487355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:24.487544 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:24.487690 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:24.487914 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 14:40:24.488037 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 14:40:24.489670 16758 tablet_service.cc:2908] Rejecting scan request for tablet 9820f8d125cc41f7ae4d57bcd970cd8b: Uninitialized: clean time has not yet been initialized
I20260810 14:40:24.491827 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 16689
I20260810 14:40:24.503499 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 16895
I20260810 14:40:24.519661 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 17108
I20260810 14:40:24.539813 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 16540
2026-08-10T14:40:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2686 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:24 dist-test-slave-rmhg krb5kdc[17391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:24 dist-test-slave-rmhg krb5kdc[17391](info): set up 2 sockets
Aug 10 14:40:24 dist-test-slave-rmhg krb5kdc[17391](info): commencing operation
krb5kdc: starting...
W20260810 14:40:25.696259 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:40:25 dist-test-slave-rmhg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:40:25.828222 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:37269
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:45009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:37269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:25.947306 17415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:25.947623 17415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:25.947683 17415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:25.951789 17415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:25.951864 17415 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 14:40:25.951889 17415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:25.951911 17415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:25.951942 17415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:25.951972 17415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:25.957579 17415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:45009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:37269
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:37269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:25.958696 17415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:25.959749 17415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:25.968333 17428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:25.968518 17415 server_base.cc:1034] running on GCE node
W20260810 14:40:25.968333 17423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:25.968350 17426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:25.969362 17415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:25.970513 17415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:25.971695 17415 hybrid_clock.cc:648] HybridClock initialized: now 1786372825971666 us; error 42 us; skew 500 ppm
Aug 10 14:40:25 dist-test-slave-rmhg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372825, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:25.974642 17415 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:40:25.975647 17415 webserver.cc:460] Webserver started at http://127.15.84.62:44971/ using document root <none> and password file <none>
I20260810 14:40:25.976367 17415 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:25.976446 17415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:25.976718 17415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:25.978976 17415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "af4442aa0fbf49c08184c4a8338fbd9a"
format_stamp: "Formatted at 2026-08-10 14:40:25 on dist-test-slave-rmhg"
server_key: "76ccabc50142e4f9bfe592e0c5fe9214"
server_key_iv: "0a341d29bb66c7e8b02497487297a141"
server_key_version: "encryptionkey@0"
I20260810 14:40:25.979575 17415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "af4442aa0fbf49c08184c4a8338fbd9a"
format_stamp: "Formatted at 2026-08-10 14:40:25 on dist-test-slave-rmhg"
server_key: "76ccabc50142e4f9bfe592e0c5fe9214"
server_key_iv: "0a341d29bb66c7e8b02497487297a141"
server_key_version: "encryptionkey@0"
I20260810 14:40:25.983726 17415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 14:40:25.986454 17438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:25.987923 17415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:40:25.988094 17415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "af4442aa0fbf49c08184c4a8338fbd9a"
format_stamp: "Formatted at 2026-08-10 14:40:25 on dist-test-slave-rmhg"
server_key: "76ccabc50142e4f9bfe592e0c5fe9214"
server_key_iv: "0a341d29bb66c7e8b02497487297a141"
server_key_version: "encryptionkey@0"
I20260810 14:40:25.988220 17415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:26.002969 17415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:26.006165 17415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:26.006371 17415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:26.014703 17415 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:37269
I20260810 14:40:26.014770 17538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:37269 every 8 connection(s)
I20260810 14:40:26.015800 17415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:26.019019 17539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:26.025404 17539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a: Bootstrap starting.
I20260810 14:40:26.026073 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 17415
I20260810 14:40:26.026196 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:26.026439 15696 external_mini_cluster.cc:1428] Setting key 5ce681ef2b68ced395cfb8caefd4b83e
I20260810 14:40:26.028427 17539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:26.029280 17539 log.cc:824] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:26.031345 17539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a: No bootstrap required, opened a new log
I20260810 14:40:26.035116 17539 raft_consensus.cc:348] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4442aa0fbf49c08184c4a8338fbd9a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37269 } }
I20260810 14:40:26.035467 17539 raft_consensus.cc:374] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 10 14:40:26 dist-test-slave-rmhg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:26.035549 17539 raft_consensus.cc:729] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af4442aa0fbf49c08184c4a8338fbd9a, State: Initialized, Role: FOLLOWER
I20260810 14:40:26.036036 17539 consensus_queue.cc:260] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4442aa0fbf49c08184c4a8338fbd9a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37269 } }
I20260810 14:40:26.036170 17539 raft_consensus.cc:388] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:26.036235 17539 raft_consensus.cc:482] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:26.036327 17539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:26.037459 17539 raft_consensus.cc:504] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4442aa0fbf49c08184c4a8338fbd9a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37269 } }
I20260810 14:40:26.037818 17539 leader_election.cc:302] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af4442aa0fbf49c08184c4a8338fbd9a; no voters:
I20260810 14:40:26.038144 17539 leader_election.cc:288] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:26.038239 17545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:26.038416 17545 raft_consensus.cc:686] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [term 1 LEADER]: Becoming Leader. State: Replica: af4442aa0fbf49c08184c4a8338fbd9a, State: Running, Role: LEADER
I20260810 14:40:26.038733 17545 consensus_queue.cc:237] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4442aa0fbf49c08184c4a8338fbd9a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37269 } }
I20260810 14:40:26.039618 17539 sys_catalog.cc:564] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:26.041664 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.027836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40530 (local address 127.15.84.62:37269)
0810 14:40:26.028352 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:26.028369 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:26.028409 (+ 40us) server_negotiation.cc:407] Connection header received
0810 14:40:26.029196 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:26.029220 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:26.029642 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:26.029988 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:26.030833 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.031907 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.032797 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.033353 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.033510 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.036182 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:26.036189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:26.036207 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:26.036246 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:26.038025 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.038562 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.038569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.038575 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.038718 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.039016 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.039020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.039022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.039506 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:26.039692 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:26.040323 (+ 631us) server_negotiation.cc:299] Negotiation successful
0810 14:40:26.040634 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":171,"threads_started":1}
I20260810 14:40:26.042658 17548 sys_catalog.cc:455] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader af4442aa0fbf49c08184c4a8338fbd9a. Latest consensus state: current_term: 1 leader_uuid: "af4442aa0fbf49c08184c4a8338fbd9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4442aa0fbf49c08184c4a8338fbd9a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37269 } } }
I20260810 14:40:26.042793 17548 sys_catalog.cc:458] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:26.043066 17546 sys_catalog.cc:455] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af4442aa0fbf49c08184c4a8338fbd9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4442aa0fbf49c08184c4a8338fbd9a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37269 } } }
I20260810 14:40:26.043159 17546 sys_catalog.cc:458] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a [sys.catalog]: This master's current role is: LEADER
W20260810 14:40:26.045222 17568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:40:26.045302 17568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:40:26.045359 17569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:26.046257 17569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:26.052371 17569 catalog_manager.cc:1347] Generated new cluster ID: 409c21a4208e425ab6f1c7afb9046da4
I20260810 14:40:26.052450 17569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:26.075169 17569 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:26.076195 17569 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:26.082593 17569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a: Generated new TSK 0
I20260810 14:40:26.083474 17569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:40:26.122401 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:37269
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:45009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:26.248592 17576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:26.248914 17576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:26.248965 17576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:26.253222 17576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:26.253314 17576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:26.253356 17576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:40:26.253389 17576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:40:26.259215 17576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:45009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.84.62:37269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:26.260408 17576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:26.261545 17576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:26.272336 17582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:26.272328 17581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:26.272394 17576 server_base.cc:1034] running on GCE node
W20260810 14:40:26.272323 17587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:26.273119 17576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:26.273936 17576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:26.275121 17576 hybrid_clock.cc:648] HybridClock initialized: now 1786372826275093 us; error 44 us; skew 500 ppm
Aug 10 14:40:26 dist-test-slave-rmhg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372826, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:26.278153 17576 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:40:26.279163 17576 webserver.cc:460] Webserver started at http://127.15.84.1:40143/ using document root <none> and password file <none>
I20260810 14:40:26.279762 17576 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:26.279829 17576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:26.280033 17576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:26.282305 17576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "7b0007133fd7410282cf09b8f212471d"
format_stamp: "Formatted at 2026-08-10 14:40:26 on dist-test-slave-rmhg"
server_key: "9d6c0802916a0d15bb9ed9892b6874cf"
server_key_iv: "32e46820218b6f00ea27a45057c369e5"
server_key_version: "encryptionkey@0"
I20260810 14:40:26.282913 17576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "7b0007133fd7410282cf09b8f212471d"
format_stamp: "Formatted at 2026-08-10 14:40:26 on dist-test-slave-rmhg"
server_key: "9d6c0802916a0d15bb9ed9892b6874cf"
server_key_iv: "32e46820218b6f00ea27a45057c369e5"
server_key_version: "encryptionkey@0"
I20260810 14:40:26.286929 17576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:40:26.289642 17597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:26.290956 17576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:26.291065 17576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "7b0007133fd7410282cf09b8f212471d"
format_stamp: "Formatted at 2026-08-10 14:40:26 on dist-test-slave-rmhg"
server_key: "9d6c0802916a0d15bb9ed9892b6874cf"
server_key_iv: "32e46820218b6f00ea27a45057c369e5"
server_key_version: "encryptionkey@0"
I20260810 14:40:26.291167 17576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:26.314251 17576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:26.317641 17576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:26.317829 17576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:26.319434 17576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:26.319496 17576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:26.319540 17576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:26.319555 17576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:40:26 dist-test-slave-rmhg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372826, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:26.330374 17576 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:44047
I20260810 14:40:26.330667 17773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:44047 every 8 connection(s)
I20260810 14:40:26.331485 17576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:40:26.333505 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.320266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:36207 (local address 127.15.84.62:37269)
0810 14:40:26.320440 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:26.320445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:26.321434 (+ 989us) server_negotiation.cc:407] Connection header received
0810 14:40:26.322503 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:26.322508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:26.322572 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:26.322685 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:26.324079 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.324646 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.326050 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.326326 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.326431 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.329480 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:26.329483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:26.329486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:26.329516 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:26.331118 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.331697 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.331700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.331701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.331786 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.332197 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.332201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.332202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.332433 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:26.332525 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:26.333179 (+ 654us) server_negotiation.cc:299] Negotiation successful
0810 14:40:26.333333 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:40:26.334327 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.320699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37269 (local address 127.15.84.1:36207)
0810 14:40:26.321253 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:26.321311 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:26.322264 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:26.322888 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:26.322899 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:26.323333 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:26.323912 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.323928 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.324780 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.324785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.325912 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.325924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.326492 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.326497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.326699 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.327355 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:26.327377 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:26.329287 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:26.331236 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:26.331242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:26.331257 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:26.331603 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:26.331874 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:26.331877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:26.331879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:26.332057 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:26.332549 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:26.332556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:26.332872 (+ 316us) client_negotiation.cc:769] Sending connection context
0810 14:40:26.333045 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 14:40:26.333439 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":121,"threads_started":1}
I20260810 14:40:26.335929 17775 heartbeater.cc:347] Connected to a master server at 127.15.84.62:37269
I20260810 14:40:26.336104 17775 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:26.336673 17775 heartbeater.cc:511] Master 127.15.84.62:37269 requested a full tablet report, sending...
I20260810 14:40:26.338528 17465 ts_manager.cc:194] Registered new tserver with Master: 7b0007133fd7410282cf09b8f212471d (127.15.84.1:44047)
I20260810 14:40:26.339957 17465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:36207
I20260810 14:40:26.341384 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 17576
I20260810 14:40:26.341500 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:40:26.341856 15696 external_mini_cluster.cc:1428] Setting key b7462228bb40273f91b4f3a301425ee5
I20260810 14:40:26.346076 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.338228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37269 (local address 127.15.84.1:39717)
0810 14:40:26.338620 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:26.338668 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:26.338811 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:26.339064 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:26.339066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:26.339295 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:26.339436 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.339444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.340282 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.340286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.340971 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.340984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.341739 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.341743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.341843 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.342445 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:26.342461 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:26.342885 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:26.344557 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:26.344560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:26.344563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:26.344808 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:26.345054 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:26.345057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:26.345058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:26.345184 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:26.345608 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:26.345610 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:26.345704 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 14:40:26.345788 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 14:40:26.345912 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":102,"threads_started":1}
I20260810 14:40:26.346144 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.338326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:39717 (local address 127.15.84.62:37269)
0810 14:40:26.338449 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:26.338454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:26.338735 (+ 281us) server_negotiation.cc:407] Connection header received
0810 14:40:26.338911 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:26.338914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:26.338960 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:26.339028 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:26.339542 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.340071 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.341092 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.341457 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.341565 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.343020 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:26.343024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:26.343026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:26.343052 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:26.344427 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.344898 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.344900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.344901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.344978 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.345261 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.345264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.345265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.345510 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:26.345616 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:26.345855 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 14:40:26.345984 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:40:26.371973 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:37269
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:45009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:26.495033 17789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:26.495390 17789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:26.495460 17789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:26.499799 17789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:26.499886 17789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:26.499929 17789 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:40:26.499963 17789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:40:26.505698 17789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:45009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.84.62:37269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:26.506817 17789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:26.507849 17789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:26.517827 17806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:26.517942 17789 server_base.cc:1034] running on GCE node
W20260810 14:40:26.517988 17801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:26.517848 17799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:26.518512 17789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:26.519234 17789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:26.520435 17789 hybrid_clock.cc:648] HybridClock initialized: now 1786372826520395 us; error 49 us; skew 500 ppm
Aug 10 14:40:26 dist-test-slave-rmhg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372826, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:26.523491 17789 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:40:26.524521 17789 webserver.cc:460] Webserver started at http://127.15.84.2:46653/ using document root <none> and password file <none>
I20260810 14:40:26.525141 17789 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:26.525260 17789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:26.525484 17789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:26.527712 17789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "a14a251f379f4b7f8577fc56f110ec6d"
format_stamp: "Formatted at 2026-08-10 14:40:26 on dist-test-slave-rmhg"
server_key: "4059478521981ec0e31be3b1fee3cdb7"
server_key_iv: "5b4c691a2cecb74356d2eada09064511"
server_key_version: "encryptionkey@0"
I20260810 14:40:26.528406 17789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "a14a251f379f4b7f8577fc56f110ec6d"
format_stamp: "Formatted at 2026-08-10 14:40:26 on dist-test-slave-rmhg"
server_key: "4059478521981ec0e31be3b1fee3cdb7"
server_key_iv: "5b4c691a2cecb74356d2eada09064511"
server_key_version: "encryptionkey@0"
I20260810 14:40:26.532982 17789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260810 14:40:26.536124 17814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:26.537649 17789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:26.537791 17789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "a14a251f379f4b7f8577fc56f110ec6d"
format_stamp: "Formatted at 2026-08-10 14:40:26 on dist-test-slave-rmhg"
server_key: "4059478521981ec0e31be3b1fee3cdb7"
server_key_iv: "5b4c691a2cecb74356d2eada09064511"
server_key_version: "encryptionkey@0"
I20260810 14:40:26.537920 17789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:26.563939 17789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:26.567219 17789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:26.567406 17789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:26.569036 17789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:26.569097 17789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:26.569353 17789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:26.569396 17789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:40:26 dist-test-slave-rmhg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372826, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:26.581627 17789 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:35677
I20260810 14:40:26.581982 17994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:35677 every 8 connection(s)
I20260810 14:40:26.582883 17789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:40:26.585500 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.570048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:54567 (local address 127.15.84.62:37269)
0810 14:40:26.570181 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:26.570185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:26.572196 (+ 2011us) server_negotiation.cc:407] Connection header received
0810 14:40:26.573561 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:26.573565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:26.573634 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:26.573729 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:26.575769 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.576423 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.577802 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.578037 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.578147 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.581237 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:26.581242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:26.581245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:26.581285 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:26.583133 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.583771 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.583775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.583777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.583875 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.584234 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.584238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.584240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.584477 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:26.584571 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:26.585177 (+ 606us) server_negotiation.cc:299] Negotiation successful
0810 14:40:26.585328 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:40:26.586377 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.571195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37269 (local address 127.15.84.2:54567)
0810 14:40:26.571940 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:26.572066 (+ 126us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:26.573303 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:26.574009 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:26.574022 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:26.574660 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:26.575545 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.575584 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.576578 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.576581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.577660 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.577674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.578215 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.578219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.578428 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.579137 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:26.579157 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:26.580969 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:26.583278 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:26.583285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:26.583302 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:26.583666 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:26.583969 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:26.583973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:26.583975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:26.584135 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:26.584592 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:26.584600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:26.584883 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 14:40:26.585077 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 14:40:26.585474 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":160,"threads_started":1}
I20260810 14:40:26.587944 17995 heartbeater.cc:347] Connected to a master server at 127.15.84.62:37269
I20260810 14:40:26.588131 17995 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:26.588779 17995 heartbeater.cc:511] Master 127.15.84.62:37269 requested a full tablet report, sending...
I20260810 14:40:26.590130 17466 ts_manager.cc:194] Registered new tserver with Master: a14a251f379f4b7f8577fc56f110ec6d (127.15.84.2:35677)
I20260810 14:40:26.590855 17466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:54567
I20260810 14:40:26.592288 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 17789
I20260810 14:40:26.592391 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:40:26.592662 15696 external_mini_cluster.cc:1428] Setting key 6a736daf0bb234eac931c99bd4c9e79d
I20260810 14:40:26.598572 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.591199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:59665 (local address 127.15.84.62:37269)
0810 14:40:26.591342 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:26.591346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:26.591361 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:40:26.591432 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:26.591434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:26.591479 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:26.591559 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:26.592143 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.592947 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.593919 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.594144 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.594238 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.595430 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:26.595434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:26.595437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:26.595461 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:26.596807 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.597246 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.597250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.597252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.597334 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.597655 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.597658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.597659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.597915 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:26.598011 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:26.598333 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 14:40:26.598447 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:40:26.598644 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.590276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37269 (local address 127.15.84.2:59665)
0810 14:40:26.590642 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:26.590658 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:26.590751 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:26.591676 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:26.591678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:26.591880 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:26.592008 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.592015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.593113 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.593117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.593795 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.593807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.594255 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.594259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.594354 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.594903 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:26.594916 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:26.595284 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:26.596920 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:26.596924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:26.596926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:26.597132 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:26.597436 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:26.597440 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:26.597442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:26.597550 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:26.598020 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:26.598034 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:26.598153 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 14:40:26.598258 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 14:40:26.598419 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":151,"threads_started":1}
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:40:26.623585 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:37269
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:45009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:26.743187 18012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:26.743507 18012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:26.743578 18012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:26.747758 18012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:26.747841 18012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:26.747897 18012 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:40:26.747946 18012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:40:26.753641 18012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:45009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.84.62:37269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:26.754802 18012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:26.755887 18012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:40:26.766454 18012 server_base.cc:1034] running on GCE node
W20260810 14:40:26.766343 18023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:26.766341 18019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:26.766341 18018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:26.767160 18012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:26.767877 18012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:26.769081 18012 hybrid_clock.cc:648] HybridClock initialized: now 1786372826769067 us; error 54 us; skew 500 ppm
Aug 10 14:40:26 dist-test-slave-rmhg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372826, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:26.772259 18012 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:40:26.773437 18012 webserver.cc:460] Webserver started at http://127.15.84.3:34665/ using document root <none> and password file <none>
I20260810 14:40:26.774072 18012 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:26.774154 18012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:26.774389 18012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:26.776691 18012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "504d41d1743142a08e4d1d6e0cb2e075"
format_stamp: "Formatted at 2026-08-10 14:40:26 on dist-test-slave-rmhg"
server_key: "291f361590de40b3b0d682e1e62d0c34"
server_key_iv: "b1b248437897a7c2db7397c1d8b71ba5"
server_key_version: "encryptionkey@0"
I20260810 14:40:26.777338 18012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "504d41d1743142a08e4d1d6e0cb2e075"
format_stamp: "Formatted at 2026-08-10 14:40:26 on dist-test-slave-rmhg"
server_key: "291f361590de40b3b0d682e1e62d0c34"
server_key_iv: "b1b248437897a7c2db7397c1d8b71ba5"
server_key_version: "encryptionkey@0"
I20260810 14:40:26.781581 18012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:40:26.784451 18037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:26.785954 18012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:26.786095 18012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "504d41d1743142a08e4d1d6e0cb2e075"
format_stamp: "Formatted at 2026-08-10 14:40:26 on dist-test-slave-rmhg"
server_key: "291f361590de40b3b0d682e1e62d0c34"
server_key_iv: "b1b248437897a7c2db7397c1d8b71ba5"
server_key_version: "encryptionkey@0"
I20260810 14:40:26.786211 18012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:26.806813 18012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:26.810020 18012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:26.810221 18012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:26.811818 18012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:26.811877 18012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:26.811934 18012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:26.811964 18012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:40:26 dist-test-slave-rmhg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372826, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:26.822973 18012 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:46173
I20260810 14:40:26.823038 18213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:46173 every 8 connection(s)
I20260810 14:40:26.824193 18012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:40:26.825973 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.812711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:35823 (local address 127.15.84.62:37269)
0810 14:40:26.812872 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:26.812877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:26.813908 (+ 1031us) server_negotiation.cc:407] Connection header received
0810 14:40:26.815235 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:26.815240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:26.815307 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:26.815404 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:26.816902 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.817512 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.818864 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.819232 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.819361 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.822128 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:26.822132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:26.822135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:26.822167 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:26.823852 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.824376 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.824380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.824382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.824480 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.824880 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.824884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.824885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.825110 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:26.825237 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:26.825735 (+ 498us) server_negotiation.cc:299] Negotiation successful
0810 14:40:26.825850 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:40:26.826865 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.813121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37269 (local address 127.15.84.3:35823)
0810 14:40:26.813699 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:26.813766 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:26.814932 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:26.815642 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:26.815650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:26.816099 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:26.816725 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.816741 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.817644 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.817648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.818734 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.818745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.819419 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.819424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.819654 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.820302 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:26.820319 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:26.821929 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:26.823985 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:26.823990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:26.824004 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:26.824267 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:26.824578 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:26.824582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:26.824584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:26.824767 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:26.825234 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:26.825241 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:26.825495 (+ 254us) client_negotiation.cc:769] Sending connection context
0810 14:40:26.825676 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 14:40:26.825948 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":120,"threads_started":1}
I20260810 14:40:26.828078 18214 heartbeater.cc:347] Connected to a master server at 127.15.84.62:37269
I20260810 14:40:26.828235 18214 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:26.828734 18214 heartbeater.cc:511] Master 127.15.84.62:37269 requested a full tablet report, sending...
I20260810 14:40:26.829838 17465 ts_manager.cc:194] Registered new tserver with Master: 504d41d1743142a08e4d1d6e0cb2e075 (127.15.84.3:46173)
I20260810 14:40:26.830470 17465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:35823
I20260810 14:40:26.831918 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 18012
I20260810 14:40:26.832039 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:40:26.832302 15696 external_mini_cluster.cc:1428] Setting key 03351c3fbaf46a999afca8cbcc07261e
I20260810 14:40:26.834417 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:40:26.837716 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.830349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:50979 (local address 127.15.84.62:37269)
0810 14:40:26.830467 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:26.830471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:26.830652 (+ 181us) server_negotiation.cc:407] Connection header received
0810 14:40:26.830724 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:26.830727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:26.830769 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:26.830845 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:26.831276 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.832022 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.833121 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.833386 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.833477 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.834599 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:26.834603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:26.834605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:26.834630 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:26.836039 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.836434 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.836438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.836440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.836548 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.836804 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.836807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.836809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.836999 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:26.837096 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:26.837443 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 14:40:26.837558 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:40:26.837735 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.830256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37269 (local address 127.15.84.3:50979)
0810 14:40:26.830542 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:26.830555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:26.830656 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:26.830886 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:26.830889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:26.831068 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:26.831168 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.831173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.832177 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.832180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.832992 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.833004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.833491 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.833495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.833597 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.834101 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:26.834112 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:26.834447 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:26.836135 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:26.836139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:26.836142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:26.836344 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:26.836632 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:26.836636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:26.836638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:26.836722 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:26.837097 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:26.837101 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:26.837280 (+ 179us) client_negotiation.cc:769] Sending connection context
0810 14:40:26.837374 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 14:40:26.837547 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":100,"threads_started":1}
Aug 10 14:40:26 dist-test-slave-rmhg krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:40:26 dist-test-slave-rmhg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:26.862879 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.854480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.15.84.62:37269)
0810 14:40:26.854631 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:26.854635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:26.854647 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:40:26.854697 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:26.854700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:26.854742 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:26.854823 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:26.855667 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.856227 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.857108 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.857401 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.857503 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.860112 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:26.860117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:26.860119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:26.860152 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:26.861363 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.861764 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.861767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.861768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.861842 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:26.862080 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:26.862084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:26.862085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:26.862321 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:26.862442 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:26.862632 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 14:40:26.862750 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:40:26.865907 17465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:40:26.868999 17465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:26.883450 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.877764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.15.84.2:35677)
0810 14:40:26.878219 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:26.878231 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:26.878258 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:40:26.878346 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:26.878352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:26.878601 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:26.879161 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:26.879306 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.880328 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.881702 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.882734 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.882835 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.883033 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:26.883186 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 14:40:26.883292 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":115,"threads_started":1}
I20260810 14:40:26.883440 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.877559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:35677 (local address 127.0.0.1:38218)
0810 14:40:26.877997 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:26.878056 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:26.878193 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:26.878791 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:26.878798 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:26.878823 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:26.878983 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.878991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.880441 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.880443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.881556 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.881567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.882842 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.882846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.882967 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.883008 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 14:40:26.883131 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 14:40:26.883279 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":28,"thread_start_us":111,"threads_started":1}
I20260810 14:40:26.883864 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.878375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:44047 (local address 127.0.0.1:39892)
0810 14:40:26.878988 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:26.879000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:26.879076 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:26.879659 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:26.879662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:26.879682 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:26.879824 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.879832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.881337 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.881341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.882391 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.882401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.883458 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.883462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.883561 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.883586 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:40:26.883646 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 14:40:26.883723 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260810 14:40:26.884033 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.877878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:46173 (local address 127.0.0.1:47144)
0810 14:40:26.878578 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:26.878591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:26.878676 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:26.879762 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:26.879766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:26.879780 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:26.879943 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.879951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.881303 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.881307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.882615 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.882627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.883704 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.883708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.883797 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.883818 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 14:40:26.883863 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 14:40:26.883925 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":48,"thread_start_us":102,"threads_started":1}
I20260810 14:40:26.884533 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.878801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.15.84.1:44047)
0810 14:40:26.879212 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:26.879218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:26.879240 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:40:26.879315 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:26.879320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:26.879506 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:26.879666 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:26.879961 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.881182 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.882535 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.883315 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.884054 (+ 739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.884221 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:26.884277 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 14:40:26.884380 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":91,"threads_started":1}
I20260810 14:40:26.884815 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.878440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.15.84.3:46173)
0810 14:40:26.879331 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:26.879344 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:26.879366 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:40:26.879449 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:26.879454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:26.879616 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:26.879854 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:26.880047 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.881181 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.882767 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.883572 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.884301 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.884517 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:26.884572 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 14:40:26.884672 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":115,"threads_started":1}
I20260810 14:40:26.887136 17677 tablet_service.cc:1467] Processing CreateTablet for tablet 410f1413425043e2968bb509d9f350ff (DEFAULT_TABLE table=test-table [id=c28fe05d8b004031a1070870becdac1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:26.887188 17902 tablet_service.cc:1467] Processing CreateTablet for tablet 410f1413425043e2968bb509d9f350ff (DEFAULT_TABLE table=test-table [id=c28fe05d8b004031a1070870becdac1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:26.887372 18114 tablet_service.cc:1467] Processing CreateTablet for tablet 410f1413425043e2968bb509d9f350ff (DEFAULT_TABLE table=test-table [id=c28fe05d8b004031a1070870becdac1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:26.888243 17677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410f1413425043e2968bb509d9f350ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:26.888273 18114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410f1413425043e2968bb509d9f350ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:26.888266 17902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410f1413425043e2968bb509d9f350ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:26.895538 18243 tablet_bootstrap.cc:492] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d: Bootstrap starting.
I20260810 14:40:26.897254 18245 tablet_bootstrap.cc:492] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d: Bootstrap starting.
I20260810 14:40:26.897755 18247 tablet_bootstrap.cc:492] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075: Bootstrap starting.
I20260810 14:40:26.898319 18243 tablet_bootstrap.cc:654] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:26.899286 18243 log.cc:824] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:26.900068 18247 tablet_bootstrap.cc:654] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:26.901084 18247 log.cc:824] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:26.901715 18245 tablet_bootstrap.cc:654] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:26.902207 18243 tablet_bootstrap.cc:492] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d: No bootstrap required, opened a new log
I20260810 14:40:26.902426 18243 ts_tablet_manager.cc:1397] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 14:40:26.902527 18245 log.cc:824] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:26.903393 18247 tablet_bootstrap.cc:492] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075: No bootstrap required, opened a new log
I20260810 14:40:26.903610 18247 ts_tablet_manager.cc:1397] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:40:26.904315 18245 tablet_bootstrap.cc:492] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d: No bootstrap required, opened a new log
I20260810 14:40:26.904515 18245 ts_tablet_manager.cc:1397] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 14:40:26.907436 18245 raft_consensus.cc:348] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:26.907549 18243 raft_consensus.cc:348] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:26.907549 18247 raft_consensus.cc:348] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:26.907830 18247 raft_consensus.cc:374] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:26.907855 18243 raft_consensus.cc:374] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:26.907915 18247 raft_consensus.cc:729] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504d41d1743142a08e4d1d6e0cb2e075, State: Initialized, Role: FOLLOWER
I20260810 14:40:26.907927 18243 raft_consensus.cc:729] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b0007133fd7410282cf09b8f212471d, State: Initialized, Role: FOLLOWER
I20260810 14:40:26.907998 18245 raft_consensus.cc:374] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:26.908172 18245 raft_consensus.cc:729] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14a251f379f4b7f8577fc56f110ec6d, State: Initialized, Role: FOLLOWER
I20260810 14:40:26.908514 18243 consensus_queue.cc:260] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:26.908497 18247 consensus_queue.cc:260] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:26.908633 18245 consensus_queue.cc:260] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:26.909312 18214 heartbeater.cc:503] Master 127.15.84.62:37269 was elected leader, sending a full tablet report...
I20260810 14:40:26.909655 18245 ts_tablet_manager.cc:1428] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 14:40:26.909658 18247 ts_tablet_manager.cc:1428] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 14:40:26.909657 18243 ts_tablet_manager.cc:1428] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 14:40:26.909983 17995 heartbeater.cc:503] Master 127.15.84.62:37269 was elected leader, sending a full tablet report...
I20260810 14:40:26.910507 17775 heartbeater.cc:503] Master 127.15.84.62:37269 was elected leader, sending a full tablet report...
I20260810 14:40:26.970870 18254 raft_consensus.cc:482] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:26.971132 18254 raft_consensus.cc:504] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:26.972393 18254 leader_election.cc:288] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a14a251f379f4b7f8577fc56f110ec6d (127.15.84.2:35677), 504d41d1743142a08e4d1d6e0cb2e075 (127.15.84.3:46173)
I20260810 14:40:26.977687 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.972735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:46173 (local address 127.15.84.1:33723)
0810 14:40:26.973241 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:26.973253 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:26.973373 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:26.973620 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:26.973623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:26.973638 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:26.973792 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.973800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.974602 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.974606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.976358 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.976373 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.977184 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.977190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.977292 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.977318 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:40:26.977400 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 14:40:26.977484 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":244,"thread_start_us":153,"threads_started":1}
I20260810 14:40:26.977713 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.972892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:33723 (local address 127.15.84.3:46173)
0810 14:40:26.973096 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:26.973100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:26.973346 (+ 246us) server_negotiation.cc:407] Connection header received
0810 14:40:26.973463 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:26.973466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:26.973520 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:26.973594 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:26.973896 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.974475 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.976520 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.976988 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.977086 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.977268 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:26.977439 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 14:40:26.977526 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260810 14:40:26.978505 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.972913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:42319 (local address 127.15.84.2:35677)
0810 14:40:26.973068 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:26.973074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:26.973445 (+ 371us) server_negotiation.cc:407] Connection header received
0810 14:40:26.973578 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:26.973581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:26.973660 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:26.973740 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:26.974053 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.974587 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.977359 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.977912 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.978047 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.978118 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:26.978307 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 14:40:26.978397 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:40:26.978549 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:26.972735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:35677 (local address 127.15.84.1:42319)
0810 14:40:26.973314 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:26.973329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:26.973461 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:26.973780 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:26.973783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:26.973801 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:26.973935 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.973942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.974710 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.974714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.976358 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:26.976373 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:26.978064 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:26.978067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:26.978154 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:26.978177 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:40:26.978258 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 14:40:26.978341 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":175,"thread_start_us":139,"threads_started":1}
I20260810 14:40:26.978709 18145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410f1413425043e2968bb509d9f350ff" candidate_uuid: "7b0007133fd7410282cf09b8f212471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504d41d1743142a08e4d1d6e0cb2e075" is_pre_election: true
I20260810 14:40:26.979117 18145 raft_consensus.cc:2454] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b0007133fd7410282cf09b8f212471d in term 0.
I20260810 14:40:26.979249 17930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410f1413425043e2968bb509d9f350ff" candidate_uuid: "7b0007133fd7410282cf09b8f212471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14a251f379f4b7f8577fc56f110ec6d" is_pre_election: true
I20260810 14:40:26.979569 17930 raft_consensus.cc:2454] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b0007133fd7410282cf09b8f212471d in term 0.
I20260810 14:40:26.979751 17604 leader_election.cc:302] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504d41d1743142a08e4d1d6e0cb2e075, 7b0007133fd7410282cf09b8f212471d; no voters:
I20260810 14:40:26.980082 18254 raft_consensus.cc:2781] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:40:26.980172 18254 raft_consensus.cc:482] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:40:26.980206 18254 raft_consensus.cc:3037] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:26.981416 18254 raft_consensus.cc:504] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:26.981814 18254 leader_election.cc:288] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [CANDIDATE]: Term 1 election: Requested vote from peers a14a251f379f4b7f8577fc56f110ec6d (127.15.84.2:35677), 504d41d1743142a08e4d1d6e0cb2e075 (127.15.84.3:46173)
I20260810 14:40:26.982255 18145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410f1413425043e2968bb509d9f350ff" candidate_uuid: "7b0007133fd7410282cf09b8f212471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504d41d1743142a08e4d1d6e0cb2e075"
I20260810 14:40:26.982255 17930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410f1413425043e2968bb509d9f350ff" candidate_uuid: "7b0007133fd7410282cf09b8f212471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14a251f379f4b7f8577fc56f110ec6d"
I20260810 14:40:26.982409 18145 raft_consensus.cc:3037] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:26.982415 17930 raft_consensus.cc:3037] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:26.984309 17930 raft_consensus.cc:2454] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b0007133fd7410282cf09b8f212471d in term 1.
I20260810 14:40:26.984309 18145 raft_consensus.cc:2454] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b0007133fd7410282cf09b8f212471d in term 1.
I20260810 14:40:26.984776 17600 leader_election.cc:302] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b0007133fd7410282cf09b8f212471d, a14a251f379f4b7f8577fc56f110ec6d; no voters:
I20260810 14:40:26.984979 18254 raft_consensus.cc:2781] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:26.985271 18254 raft_consensus.cc:686] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [term 1 LEADER]: Becoming Leader. State: Replica: 7b0007133fd7410282cf09b8f212471d, State: Running, Role: LEADER
I20260810 14:40:26.985654 18254 consensus_queue.cc:237] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:26.989114 17464 catalog_manager.cc:5495] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b0007133fd7410282cf09b8f212471d (127.15.84.1). New cstate: current_term: 1 leader_uuid: "7b0007133fd7410282cf09b8f212471d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260810 14:40:27.013645 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.009852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.15.84.1:44047)
0810 14:40:27.009970 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:27.009973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:27.010002 (+ 29us) server_negotiation.cc:407] Connection header received
0810 14:40:27.010215 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:27.010219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:27.010282 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:27.010375 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:27.010622 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.011136 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.012051 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.012272 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.012349 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.012708 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:27.013249 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:27.013345 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:27.013448 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 14:40:27.013518 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 14:40:27.020843 17930 raft_consensus.cc:1261] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d [term 1 FOLLOWER]: Refusing update from remote peer 7b0007133fd7410282cf09b8f212471d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:27.020870 18145 raft_consensus.cc:1261] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075 [term 1 FOLLOWER]: Refusing update from remote peer 7b0007133fd7410282cf09b8f212471d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:27.021653 18259 consensus_queue.cc:1035] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:27.021857 18254 consensus_queue.cc:1035] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [LEADER]: Connected to new peer: Peer: permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:27.031615 18266 mvcc.cc:205] Tried to move back new op lower bound from 7316983099469041664 to 7316983099336232960. Current Snapshot: MvccSnapshot[applied={T|T < 7316983099469041664}]
I20260810 14:40:27.032721 18267 mvcc.cc:205] Tried to move back new op lower bound from 7316983099469041664 to 7316983099336232960. Current Snapshot: MvccSnapshot[applied={T|T < 7316983099469041664}]
I20260810 14:40:27.044760 17466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:27.045070 17466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:27.047818 17466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a: Sending DeleteTablet for 3 replicas of tablet 410f1413425043e2968bb509d9f350ff
I20260810 14:40:27.048730 17902 tablet_service.cc:1514] Processing DeleteTablet for tablet 410f1413425043e2968bb509d9f350ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:27 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:38218
I20260810 14:40:27.048754 17677 tablet_service.cc:1514] Processing DeleteTablet for tablet 410f1413425043e2968bb509d9f350ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:27 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:39892
I20260810 14:40:27.049039 18114 tablet_service.cc:1514] Processing DeleteTablet for tablet 410f1413425043e2968bb509d9f350ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:27 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:47144
I20260810 14:40:27.049316 18281 tablet_replica.cc:330] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d: stopping tablet replica
I20260810 14:40:27.049422 18282 tablet_replica.cc:330] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d: stopping tablet replica
I20260810 14:40:27.049692 18282 raft_consensus.cc:2229] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:40:27.049692 18281 raft_consensus.cc:2229] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:27.050060 18281 raft_consensus.cc:2258] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:27.050072 17466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 14:40:27.050199 18282 raft_consensus.cc:2258] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 14:40:27.050529 17466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:27.051703 18281 ts_tablet_manager.cc:1905] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:27.051703 18282 ts_tablet_manager.cc:1905] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:27.054773 18281 ts_tablet_manager.cc:1918] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:27.054773 18282 ts_tablet_manager.cc:1918] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:27.054906 18281 log.cc:1196] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/410f1413425043e2968bb509d9f350ff
I20260810 14:40:27.054906 18282 log.cc:1196] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/wal/wals/410f1413425043e2968bb509d9f350ff
I20260810 14:40:27.055248 18282 ts_tablet_manager.cc:1939] T 410f1413425043e2968bb509d9f350ff P 7b0007133fd7410282cf09b8f212471d: Deleting consensus metadata
I20260810 14:40:27.055248 18281 ts_tablet_manager.cc:1939] T 410f1413425043e2968bb509d9f350ff P a14a251f379f4b7f8577fc56f110ec6d: Deleting consensus metadata
I20260810 14:40:27.056624 17441 catalog_manager.cc:4841] TS a14a251f379f4b7f8577fc56f110ec6d (127.15.84.2:35677): tablet 410f1413425043e2968bb509d9f350ff (table test-table [id=c28fe05d8b004031a1070870becdac1e]) successfully deleted
I20260810 14:40:27.056818 17441 catalog_manager.cc:4841] TS 7b0007133fd7410282cf09b8f212471d (127.15.84.1:44047): tablet 410f1413425043e2968bb509d9f350ff (table test-table [id=c28fe05d8b004031a1070870becdac1e]) successfully deleted
I20260810 14:40:27.057090 18283 tablet_replica.cc:330] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075: stopping tablet replica
I20260810 14:40:27.057544 18283 raft_consensus.cc:2229] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:27.057518 17902 tablet_service.cc:1467] Processing CreateTablet for tablet 9f395100751c449b8a0f05493373009d (DEFAULT_TABLE table=test-table [id=5c4d84129e934b9ab866a676cfb9fe20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:27.057790 17902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f395100751c449b8a0f05493373009d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:27.057770 18114 tablet_service.cc:1467] Processing CreateTablet for tablet 9f395100751c449b8a0f05493373009d (DEFAULT_TABLE table=test-table [id=5c4d84129e934b9ab866a676cfb9fe20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:27.057914 18283 raft_consensus.cc:2258] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:27.058049 18114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f395100751c449b8a0f05493373009d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:27.059065 17677 tablet_service.cc:1467] Processing CreateTablet for tablet 9f395100751c449b8a0f05493373009d (DEFAULT_TABLE table=test-table [id=5c4d84129e934b9ab866a676cfb9fe20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:27.059322 17677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f395100751c449b8a0f05493373009d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:27.060024 18283 ts_tablet_manager.cc:1905] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:27.060273 18245 tablet_bootstrap.cc:492] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d: Bootstrap starting.
I20260810 14:40:27.061273 18245 tablet_bootstrap.cc:654] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:27.061753 18243 tablet_bootstrap.cc:492] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d: Bootstrap starting.
I20260810 14:40:27.061831 18247 tablet_bootstrap.cc:492] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075: Bootstrap starting.
I20260810 14:40:27.062538 18283 ts_tablet_manager.cc:1918] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:27.062626 18283 log.cc:1196] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/410f1413425043e2968bb509d9f350ff
I20260810 14:40:27.062750 18243 tablet_bootstrap.cc:654] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:27.062794 18247 tablet_bootstrap.cc:654] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:27.062836 18245 tablet_bootstrap.cc:492] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d: No bootstrap required, opened a new log
I20260810 14:40:27.062917 18245 ts_tablet_manager.cc:1397] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 14:40:27.062934 18283 ts_tablet_manager.cc:1939] T 410f1413425043e2968bb509d9f350ff P 504d41d1743142a08e4d1d6e0cb2e075: Deleting consensus metadata
I20260810 14:40:27.063405 18245 raft_consensus.cc:348] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.063516 18245 raft_consensus.cc:374] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:27.063546 18245 raft_consensus.cc:729] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14a251f379f4b7f8577fc56f110ec6d, State: Initialized, Role: FOLLOWER
I20260810 14:40:27.063678 18245 consensus_queue.cc:260] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.063925 18245 ts_tablet_manager.cc:1428] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:40:27.064327 17446 catalog_manager.cc:4841] TS 504d41d1743142a08e4d1d6e0cb2e075 (127.15.84.3:46173): tablet 410f1413425043e2968bb509d9f350ff (table test-table [id=c28fe05d8b004031a1070870becdac1e]) successfully deleted
I20260810 14:40:27.065335 18247 tablet_bootstrap.cc:492] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075: No bootstrap required, opened a new log
I20260810 14:40:27.065335 18243 tablet_bootstrap.cc:492] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d: No bootstrap required, opened a new log
I20260810 14:40:27.065431 18243 ts_tablet_manager.cc:1397] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 14:40:27.065431 18247 ts_tablet_manager.cc:1397] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 14:40:27.065891 18247 raft_consensus.cc:348] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.065902 18243 raft_consensus.cc:348] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.066015 18247 raft_consensus.cc:374] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:27.066015 18243 raft_consensus.cc:374] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:27.066048 18243 raft_consensus.cc:729] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b0007133fd7410282cf09b8f212471d, State: Initialized, Role: FOLLOWER
I20260810 14:40:27.066047 18247 raft_consensus.cc:729] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504d41d1743142a08e4d1d6e0cb2e075, State: Initialized, Role: FOLLOWER
I20260810 14:40:27.066174 18247 consensus_queue.cc:260] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.066174 18243 consensus_queue.cc:260] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.066423 18247 ts_tablet_manager.cc:1428] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:27.066426 18243 ts_tablet_manager.cc:1428] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260810 14:40:27.075234 18215 tablet.cc:2369] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:40:27.082942 17776 tablet.cc:2369] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:40:27.084138 17998 tablet.cc:2369] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:40:27.264618 18253 raft_consensus.cc:482] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:27.264792 18253 raft_consensus.cc:504] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.266007 18253 leader_election.cc:288] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b0007133fd7410282cf09b8f212471d (127.15.84.1:44047), a14a251f379f4b7f8577fc56f110ec6d (127.15.84.2:35677)
I20260810 14:40:27.270159 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.266266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:44047 (local address 127.15.84.3:48423)
0810 14:40:27.266401 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:27.266418 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:27.266541 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:27.266900 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:27.266904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:27.266927 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:27.267074 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:27.267081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.267943 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.267947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:27.268815 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:27.268826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.269716 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.269720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:27.269829 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.269858 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 14:40:27.269931 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 14:40:27.270010 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"spinlock_wait_cycles":23296}
I20260810 14:40:27.270159 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.266390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:48423 (local address 127.15.84.1:44047)
0810 14:40:27.266527 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:27.266535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:27.266555 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:40:27.266656 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:27.266660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:27.266714 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:27.266808 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:27.267199 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.267831 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.268945 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.269589 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.269671 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.269732 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:27.269972 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 14:40:27.270055 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:40:27.270712 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.266783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:52631 (local address 127.15.84.2:35677)
0810 14:40:27.266911 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:27.266915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:27.266931 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:40:27.267081 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:27.267084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:27.267131 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:27.267202 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:27.267473 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.267999 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.269087 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.270132 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.270226 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.270286 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:27.270527 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 14:40:27.270608 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:40:27.270722 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.266584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:35677 (local address 127.15.84.3:52631)
0810 14:40:27.266855 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:27.266870 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:27.266981 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:27.267226 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:27.267228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:27.267245 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:27.267378 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:27.267384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.268106 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.268108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:27.268963 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:27.268974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.270262 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.270265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:27.270397 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.270423 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:40:27.270496 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 14:40:27.270578 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":113,"threads_started":1}
I20260810 14:40:27.270757 17704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f395100751c449b8a0f05493373009d" candidate_uuid: "504d41d1743142a08e4d1d6e0cb2e075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b0007133fd7410282cf09b8f212471d" is_pre_election: true
I20260810 14:40:27.271065 17704 raft_consensus.cc:2454] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 504d41d1743142a08e4d1d6e0cb2e075 in term 0.
I20260810 14:40:27.271270 17930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f395100751c449b8a0f05493373009d" candidate_uuid: "504d41d1743142a08e4d1d6e0cb2e075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14a251f379f4b7f8577fc56f110ec6d" is_pre_election: true
I20260810 14:40:27.271404 17930 raft_consensus.cc:2454] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 504d41d1743142a08e4d1d6e0cb2e075 in term 0.
I20260810 14:40:27.271512 18040 leader_election.cc:302] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504d41d1743142a08e4d1d6e0cb2e075, 7b0007133fd7410282cf09b8f212471d; no voters:
I20260810 14:40:27.271806 18253 raft_consensus.cc:2781] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:40:27.271864 18253 raft_consensus.cc:482] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:40:27.271893 18253 raft_consensus.cc:3037] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:27.272976 18253 raft_consensus.cc:504] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.273361 18253 leader_election.cc:288] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [CANDIDATE]: Term 1 election: Requested vote from peers 7b0007133fd7410282cf09b8f212471d (127.15.84.1:44047), a14a251f379f4b7f8577fc56f110ec6d (127.15.84.2:35677)
I20260810 14:40:27.273845 17704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f395100751c449b8a0f05493373009d" candidate_uuid: "504d41d1743142a08e4d1d6e0cb2e075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b0007133fd7410282cf09b8f212471d"
I20260810 14:40:27.273875 17930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f395100751c449b8a0f05493373009d" candidate_uuid: "504d41d1743142a08e4d1d6e0cb2e075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14a251f379f4b7f8577fc56f110ec6d"
I20260810 14:40:27.273965 17704 raft_consensus.cc:3037] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:27.274012 17930 raft_consensus.cc:3037] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:27.274943 17704 raft_consensus.cc:2454] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504d41d1743142a08e4d1d6e0cb2e075 in term 1.
I20260810 14:40:27.274958 17930 raft_consensus.cc:2454] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504d41d1743142a08e4d1d6e0cb2e075 in term 1.
I20260810 14:40:27.275264 18040 leader_election.cc:302] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504d41d1743142a08e4d1d6e0cb2e075, 7b0007133fd7410282cf09b8f212471d; no voters:
I20260810 14:40:27.275446 18253 raft_consensus.cc:2781] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:27.275686 18253 raft_consensus.cc:686] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [term 1 LEADER]: Becoming Leader. State: Replica: 504d41d1743142a08e4d1d6e0cb2e075, State: Running, Role: LEADER
I20260810 14:40:27.275961 18253 consensus_queue.cc:237] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.278294 17466 catalog_manager.cc:5495] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 reported cstate change: term changed from 0 to 1, leader changed from <none> to 504d41d1743142a08e4d1d6e0cb2e075 (127.15.84.3). New cstate: current_term: 1 leader_uuid: "504d41d1743142a08e4d1d6e0cb2e075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } health_report { overall_health: HEALTHY } } }
Aug 10 14:40:27 dist-test-slave-rmhg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372825, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:27.348255 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.339202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37269 (local address 127.0.0.1:40540)
0810 14:40:27.339520 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:27.339533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:27.339672 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:27.339897 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:27.339899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:27.340352 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:27.340466 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:27.340474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.341393 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.341397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:27.342128 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:27.342140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.342662 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.342666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:27.342753 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.343400 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:27.343413 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:27.345071 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:27.346696 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:27.346708 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:27.346716 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:27.346943 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:27.347197 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:27.347200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:27.347202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:27.347277 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:27.347652 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:27.347659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:27.347772 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 14:40:27.347873 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 14:40:27.348058 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":134,"threads_started":1}
I20260810 14:40:27.348254 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.339289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40540 (local address 127.15.84.62:37269)
0810 14:40:27.339415 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:27.339420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:27.339608 (+ 188us) server_negotiation.cc:407] Connection header received
0810 14:40:27.339756 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:27.339759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:27.339798 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:27.339865 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:27.340627 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.341292 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.342240 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.342514 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.342617 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.345234 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:27.345238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:27.345241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:27.345268 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:27.346589 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:27.347033 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:27.347037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:27.347039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:27.347118 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:27.347351 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:27.347354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:27.347356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:27.347552 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:27.347638 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:27.347932 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 14:40:27.348099 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":41}
I20260810 14:40:27.352411 17466 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:40540:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 14:40:27.353193 17466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:27.358140 17677 tablet_service.cc:1467] Processing CreateTablet for tablet eb7cdea715254b5aa806cdff92a46f3a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=30904cfb64434b82a32bddc31dc5b169]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:40:27.358196 18114 tablet_service.cc:1467] Processing CreateTablet for tablet eb7cdea715254b5aa806cdff92a46f3a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=30904cfb64434b82a32bddc31dc5b169]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:40:27.358173 17902 tablet_service.cc:1467] Processing CreateTablet for tablet eb7cdea715254b5aa806cdff92a46f3a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=30904cfb64434b82a32bddc31dc5b169]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:40:27.358592 17902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7cdea715254b5aa806cdff92a46f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:27.358590 18114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7cdea715254b5aa806cdff92a46f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:27.358618 17677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7cdea715254b5aa806cdff92a46f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:27.361356 18243 tablet_bootstrap.cc:492] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d: Bootstrap starting.
I20260810 14:40:27.361620 18247 tablet_bootstrap.cc:492] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075: Bootstrap starting.
I20260810 14:40:27.361620 18245 tablet_bootstrap.cc:492] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d: Bootstrap starting.
I20260810 14:40:27.362344 18243 tablet_bootstrap.cc:654] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:27.362603 18247 tablet_bootstrap.cc:654] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:27.362603 18245 tablet_bootstrap.cc:654] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:27.363684 18243 tablet_bootstrap.cc:492] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d: No bootstrap required, opened a new log
I20260810 14:40:27.363780 18243 ts_tablet_manager.cc:1397] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:40:27.363982 18247 tablet_bootstrap.cc:492] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075: No bootstrap required, opened a new log
I20260810 14:40:27.364074 18247 ts_tablet_manager.cc:1397] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:40:27.364089 18245 tablet_bootstrap.cc:492] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d: No bootstrap required, opened a new log
I20260810 14:40:27.364156 18245 ts_tablet_manager.cc:1397] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 14:40:27.364241 18243 raft_consensus.cc:348] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.364372 18243 raft_consensus.cc:374] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:27.364410 18243 raft_consensus.cc:729] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b0007133fd7410282cf09b8f212471d, State: Initialized, Role: FOLLOWER
I20260810 14:40:27.364571 18243 consensus_queue.cc:260] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.364554 18247 raft_consensus.cc:348] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.364575 18245 raft_consensus.cc:348] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.364671 18247 raft_consensus.cc:374] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:27.364679 18245 raft_consensus.cc:374] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:27.364702 18247 raft_consensus.cc:729] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504d41d1743142a08e4d1d6e0cb2e075, State: Initialized, Role: FOLLOWER
I20260810 14:40:27.364722 18245 raft_consensus.cc:729] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14a251f379f4b7f8577fc56f110ec6d, State: Initialized, Role: FOLLOWER
I20260810 14:40:27.364825 18247 consensus_queue.cc:260] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.364830 18245 consensus_queue.cc:260] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.364840 18276 tablet_replica.cc:439] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } } }
I20260810 14:40:27.364980 18276 tablet_replica.cc:442] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:40:27.365096 18247 ts_tablet_manager.cc:1428] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:27.365096 18245 ts_tablet_manager.cc:1428] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:27.365368 18253 tablet_replica.cc:439] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } } }
I20260810 14:40:27.365512 18253 tablet_replica.cc:442] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:40:27.365831 18243 ts_tablet_manager.cc:1428] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:40:27.365777 18255 tablet_replica.cc:439] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } } }
I20260810 14:40:27.365918 18255 tablet_replica.cc:442] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:40:27.494180 18276 raft_consensus.cc:482] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:27.494354 18276 raft_consensus.cc:504] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.494833 18276 leader_election.cc:288] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a14a251f379f4b7f8577fc56f110ec6d (127.15.84.2:35677), 504d41d1743142a08e4d1d6e0cb2e075 (127.15.84.3:46173)
I20260810 14:40:27.495398 17930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7cdea715254b5aa806cdff92a46f3a" candidate_uuid: "7b0007133fd7410282cf09b8f212471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14a251f379f4b7f8577fc56f110ec6d" is_pre_election: true
I20260810 14:40:27.495411 18145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7cdea715254b5aa806cdff92a46f3a" candidate_uuid: "7b0007133fd7410282cf09b8f212471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504d41d1743142a08e4d1d6e0cb2e075" is_pre_election: true
I20260810 14:40:27.495565 17930 raft_consensus.cc:2454] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b0007133fd7410282cf09b8f212471d in term 0.
I20260810 14:40:27.495584 18145 raft_consensus.cc:2454] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b0007133fd7410282cf09b8f212471d in term 0.
I20260810 14:40:27.495926 17604 leader_election.cc:302] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 504d41d1743142a08e4d1d6e0cb2e075, 7b0007133fd7410282cf09b8f212471d, a14a251f379f4b7f8577fc56f110ec6d; no voters:
I20260810 14:40:27.496145 18276 raft_consensus.cc:2781] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:40:27.496201 18276 raft_consensus.cc:482] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:40:27.496227 18276 raft_consensus.cc:3037] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:27.497316 18276 raft_consensus.cc:504] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.497676 18276 leader_election.cc:288] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [CANDIDATE]: Term 1 election: Requested vote from peers a14a251f379f4b7f8577fc56f110ec6d (127.15.84.2:35677), 504d41d1743142a08e4d1d6e0cb2e075 (127.15.84.3:46173)
I20260810 14:40:27.498116 17930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7cdea715254b5aa806cdff92a46f3a" candidate_uuid: "7b0007133fd7410282cf09b8f212471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14a251f379f4b7f8577fc56f110ec6d"
I20260810 14:40:27.498116 18145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7cdea715254b5aa806cdff92a46f3a" candidate_uuid: "7b0007133fd7410282cf09b8f212471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504d41d1743142a08e4d1d6e0cb2e075"
I20260810 14:40:27.498255 17930 raft_consensus.cc:3037] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:27.498255 18145 raft_consensus.cc:3037] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:27.499378 18145 raft_consensus.cc:2454] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b0007133fd7410282cf09b8f212471d in term 1.
I20260810 14:40:27.499377 17930 raft_consensus.cc:2454] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b0007133fd7410282cf09b8f212471d in term 1.
I20260810 14:40:27.499704 17600 leader_election.cc:302] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 504d41d1743142a08e4d1d6e0cb2e075, 7b0007133fd7410282cf09b8f212471d, a14a251f379f4b7f8577fc56f110ec6d; no voters:
I20260810 14:40:27.499891 18276 raft_consensus.cc:2781] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:27.500015 18276 raft_consensus.cc:686] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [term 1 LEADER]: Becoming Leader. State: Replica: 7b0007133fd7410282cf09b8f212471d, State: Running, Role: LEADER
I20260810 14:40:27.500174 18276 consensus_queue.cc:237] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } }
I20260810 14:40:27.500870 18254 tablet_replica.cc:439] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d: TxnStatusTablet state changed. Reason: New leader 7b0007133fd7410282cf09b8f212471d. Latest consensus state: current_term: 1 leader_uuid: "7b0007133fd7410282cf09b8f212471d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } } }
I20260810 14:40:27.500969 18254 tablet_replica.cc:442] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:40:27.501312 18306 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 14:40:27.501596 18306 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 14:40:27.502022 17466 catalog_manager.cc:5495] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b0007133fd7410282cf09b8f212471d (127.15.84.1). New cstate: current_term: 1 leader_uuid: "7b0007133fd7410282cf09b8f212471d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260810 14:40:27.520260 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.517125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.15.84.1:44047)
0810 14:40:27.517282 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:27.517287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:27.517302 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:40:27.517424 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:27.517428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:27.517478 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:27.517543 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:27.517792 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.518315 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.519163 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.519392 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.519479 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.519782 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:27.519874 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:27.519939 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:27.520062 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 14:40:27.520119 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 14:40:27.520278 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.516988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:44047 (local address 127.0.0.1:39900)
0810 14:40:27.517165 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:27.517182 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:27.517297 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:27.517571 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:27.517574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:27.517584 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:40:27.517694 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:27.517699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.518423 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.518426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:27.519042 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:27.519050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.519490 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.519493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:27.519582 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.519631 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:40:27.519953 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:40:27.519971 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 14:40:27.520042 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 14:40:27.520163 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 14:40:27.522888 17930 raft_consensus.cc:1261] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d [term 1 FOLLOWER]: Refusing update from remote peer 7b0007133fd7410282cf09b8f212471d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:27.522858 18145 raft_consensus.cc:1261] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075 [term 1 FOLLOWER]: Refusing update from remote peer 7b0007133fd7410282cf09b8f212471d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:27.523325 18254 consensus_queue.cc:1035] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [LEADER]: Connected to new peer: Peer: permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:27.523483 18276 consensus_queue.cc:1035] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:27.525020 18255 tablet_replica.cc:439] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d: TxnStatusTablet state changed. Reason: New leader 7b0007133fd7410282cf09b8f212471d. Latest consensus state: current_term: 1 leader_uuid: "7b0007133fd7410282cf09b8f212471d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } } }
I20260810 14:40:27.525173 18255 tablet_replica.cc:442] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:40:27.526021 18253 tablet_replica.cc:439] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075: TxnStatusTablet state changed. Reason: New leader 7b0007133fd7410282cf09b8f212471d. Latest consensus state: current_term: 1 leader_uuid: "7b0007133fd7410282cf09b8f212471d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } } }
I20260810 14:40:27.526154 18253 tablet_replica.cc:442] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:40:27.527772 18259 tablet_replica.cc:439] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b0007133fd7410282cf09b8f212471d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } } }
I20260810 14:40:27.527804 18276 tablet_replica.cc:439] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b0007133fd7410282cf09b8f212471d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } } }
I20260810 14:40:27.527889 18259 tablet_replica.cc:442] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:40:27.527910 18276 tablet_replica.cc:442] T eb7cdea715254b5aa806cdff92a46f3a P 7b0007133fd7410282cf09b8f212471d: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:40:27.527979 18263 mvcc.cc:205] Tried to move back new op lower bound from 7316983101529403392 to 7316983101442093056. Current Snapshot: MvccSnapshot[applied={T|T < 7316983101529403392}]
I20260810 14:40:27.529780 18253 tablet_replica.cc:439] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b0007133fd7410282cf09b8f212471d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } } }
I20260810 14:40:27.529914 18253 tablet_replica.cc:442] T eb7cdea715254b5aa806cdff92a46f3a P 504d41d1743142a08e4d1d6e0cb2e075: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:40:27.530129 18255 tablet_replica.cc:439] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b0007133fd7410282cf09b8f212471d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 } } peers { permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 } } peers { permanent_uuid: "504d41d1743142a08e4d1d6e0cb2e075" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46173 } } }
I20260810 14:40:27.530248 18255 tablet_replica.cc:442] T eb7cdea715254b5aa806cdff92a46f3a P a14a251f379f4b7f8577fc56f110ec6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:40:27.541672 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.533869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.15.84.3:46173)
0810 14:40:27.537016 (+ 3147us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:27.537021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:27.537042 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:40:27.537183 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:27.537187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:27.537266 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:27.537404 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:27.537717 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.538556 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.539866 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.540115 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.540244 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.540653 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:27.541243 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:27.541370 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:27.541408 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 14:40:27.541473 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3048,"thread_start_us":116,"threads_started":1}
I20260810 14:40:27.554824 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.550013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:58719 (local address 127.15.84.1:44047)
0810 14:40:27.550193 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:27.550197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:27.550307 (+ 110us) server_negotiation.cc:407] Connection header received
0810 14:40:27.550515 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:27.550518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:27.550577 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:27.550648 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:27.550916 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.551431 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.553696 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.553938 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.554026 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.554383 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:27.554475 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:27.554549 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:27.554640 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 14:40:27.554702 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260810 14:40:27.554911 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.549871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:44047 (local address 127.15.84.3:58719)
0810 14:40:27.550223 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:27.550238 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:27.550357 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:27.550674 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:27.550677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:27.550685 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:40:27.550815 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:27.550821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.551528 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.551531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:27.553559 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:27.553574 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.554048 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.554052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:27.554143 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.554209 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:40:27.554562 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:40:27.554567 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:40:27.554629 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 14:40:27.554733 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":123,"threads_started":1}
I20260810 14:40:27.563163 17930 raft_consensus.cc:1261] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d [term 1 FOLLOWER]: Refusing update from remote peer 504d41d1743142a08e4d1d6e0cb2e075: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:27.563347 17704 raft_consensus.cc:1261] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d [term 1 FOLLOWER]: Refusing update from remote peer 504d41d1743142a08e4d1d6e0cb2e075: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:27.563747 18253 consensus_queue.cc:1035] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14a251f379f4b7f8577fc56f110ec6d" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:27.563956 18290 consensus_queue.cc:1035] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b0007133fd7410282cf09b8f212471d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:27.593891 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.588785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:46173 (local address 127.15.84.1:56111)
0810 14:40:27.589272 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:27.589290 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:27.589414 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:27.589722 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:27.589725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:27.589733 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:40:27.589855 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:27.589861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.590959 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.590963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:27.592090 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:27.592105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.592889 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.592893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:27.592989 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.593057 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:40:27.593541 (+ 484us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:40:27.593547 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:40:27.593604 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 14:40:27.593717 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":127,"threads_started":1}
I20260810 14:40:27.594352 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.589052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:56111 (local address 127.15.84.3:46173)
0810 14:40:27.589222 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:27.589226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:27.589382 (+ 156us) server_negotiation.cc:407] Connection header received
0810 14:40:27.589546 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:27.589549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:27.589610 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:27.589702 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:27.589957 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.590812 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.592319 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.592692 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.592793 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.593317 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:27.593423 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:27.593471 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:27.594154 (+ 683us) server_negotiation.cc:299] Negotiation successful
0810 14:40:27.594224 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:40:27.627722 17464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:27.627916 17464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:27.630358 17464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af4442aa0fbf49c08184c4a8338fbd9a: Sending DeleteTablet for 3 replicas of tablet 9f395100751c449b8a0f05493373009d
I20260810 14:40:27.631182 17677 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f395100751c449b8a0f05493373009d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:27 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:39892
I20260810 14:40:27.631361 17902 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f395100751c449b8a0f05493373009d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:27 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:38218
I20260810 14:40:27.631788 18114 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f395100751c449b8a0f05493373009d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:27 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:47144
I20260810 14:40:27.632221 18332 tablet_replica.cc:330] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075: stopping tablet replica
I20260810 14:40:27.632364 18332 raft_consensus.cc:2229] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:40:27.632733 18332 raft_consensus.cc:2258] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:27.633388 18328 tablet_replica.cc:330] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d: stopping tablet replica
I20260810 14:40:27.633558 18332 ts_tablet_manager.cc:1905] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:27.633665 18328 raft_consensus.cc:2229] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:27.633867 18328 raft_consensus.cc:2258] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:27.634979 18328 ts_tablet_manager.cc:1905] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:27.638613 18328 ts_tablet_manager.cc:1918] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:40:27.638751 18328 log.cc:1196] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-0/wal/wals/9f395100751c449b8a0f05493373009d
I20260810 14:40:27.639046 18328 ts_tablet_manager.cc:1939] T 9f395100751c449b8a0f05493373009d P 7b0007133fd7410282cf09b8f212471d: Deleting consensus metadata
I20260810 14:40:27.639962 18332 ts_tablet_manager.cc:1918] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:40:27.640064 18332 log.cc:1196] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/9f395100751c449b8a0f05493373009d
I20260810 14:40:27.640097 17441 catalog_manager.cc:4841] TS 7b0007133fd7410282cf09b8f212471d (127.15.84.1:44047): tablet 9f395100751c449b8a0f05493373009d (table test-table [id=5c4d84129e934b9ab866a676cfb9fe20]) successfully deleted
I20260810 14:40:27.640344 18332 ts_tablet_manager.cc:1939] T 9f395100751c449b8a0f05493373009d P 504d41d1743142a08e4d1d6e0cb2e075: Deleting consensus metadata
I20260810 14:40:27.640707 18329 tablet_replica.cc:330] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d: stopping tablet replica
I20260810 14:40:27.640941 18329 raft_consensus.cc:2229] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:27.641191 18329 raft_consensus.cc:2258] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:27.641445 17446 catalog_manager.cc:4841] TS 504d41d1743142a08e4d1d6e0cb2e075 (127.15.84.3:46173): tablet 9f395100751c449b8a0f05493373009d (table test-table [id=5c4d84129e934b9ab866a676cfb9fe20]) successfully deleted
I20260810 14:40:27.642555 18329 ts_tablet_manager.cc:1905] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:27.647895 18329 ts_tablet_manager.cc:1918] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:40:27.648073 18329 log.cc:1196] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/9f395100751c449b8a0f05493373009d
I20260810 14:40:27.648473 18329 ts_tablet_manager.cc:1939] T 9f395100751c449b8a0f05493373009d P a14a251f379f4b7f8577fc56f110ec6d: Deleting consensus metadata
Aug 10 14:40:27 dist-test-slave-rmhg krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.84.1@KRBTEST.COM
I20260810 14:40:27.649813 17441 catalog_manager.cc:4841] TS a14a251f379f4b7f8577fc56f110ec6d (127.15.84.2:35677): tablet 9f395100751c449b8a0f05493373009d (table test-table [id=5c4d84129e934b9ab866a676cfb9fe20]) successfully deleted
I20260810 14:40:27.657922 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.634466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39906 (local address 127.15.84.1:44047)
0810 14:40:27.634613 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:27.634618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:27.634928 (+ 310us) server_negotiation.cc:407] Connection header received
0810 14:40:27.635066 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:27.635070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:27.635118 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:27.635185 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:27.635936 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.636541 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.642830 (+ 6289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.643384 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.643547 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.650463 (+ 6916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:27.650491 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:27.650522 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:27.650581 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:27.654226 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:27.655348 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:27.655364 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:27.655366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:27.655572 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:27.656079 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:27.656083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:27.656085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:27.656670 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:27.657081 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:27.657415 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 14:40:27.657616 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 14:40:27.658842 17765 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39906
I20260810 14:40:27.678012 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:27.661951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.15.84.62:37269)
0810 14:40:27.662160 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:27.662168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:27.662843 (+ 675us) server_negotiation.cc:407] Connection header received
0810 14:40:27.663032 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:27.663036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:27.663135 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:27.663278 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:27.664583 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.665802 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.667428 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:27.667931 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:27.668075 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:27.671129 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:27.671135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:27.671138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:27.671198 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:27.674339 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:27.675118 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:27.675123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:27.675126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:27.675296 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:27.675748 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:27.675751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:27.675753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:27.676132 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:27.676436 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:27.677549 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0810 14:40:27.677750 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260810 14:40:27.678958 17464 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40548
I20260810 14:40:27.681814 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 17576
I20260810 14:40:27.700665 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 17789
I20260810 14:40:27.733675 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 18012
I20260810 14:40:27.757388 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 17415
2026-08-10T14:40:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3221 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:27 dist-test-slave-rmhg krb5kdc[18339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:27 dist-test-slave-rmhg krb5kdc[18339](info): set up 2 sockets
Aug 10 14:40:27 dist-test-slave-rmhg krb5kdc[18339](info): commencing operation
krb5kdc: starting...
W20260810 14:40:29.468744 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.673s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:40:29 dist-test-slave-rmhg krb5kdc[18339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:40:29.650364 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:41501
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:41501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:29.769286 18358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:29.769789 18358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:29.769886 18358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:29.774120 18358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:29.774204 18358 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 14:40:29.774228 18358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:29.774250 18358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:29.774272 18358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:29.774302 18358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:29.780193 18358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:41501
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:41501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:29.781502 18358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:29.782625 18358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:29.791858 18363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:29.791870 18365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:29.791919 18358 server_base.cc:1034] running on GCE node
W20260810 14:40:29.791877 18368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:29.792728 18358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:29.794229 18358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:29.795449 18358 hybrid_clock.cc:648] HybridClock initialized: now 1786372829795417 us; error 61 us; skew 500 ppm
Aug 10 14:40:29 dist-test-slave-rmhg krb5kdc[18339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372829, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:29.802271 18358 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:40:29.803325 18358 webserver.cc:460] Webserver started at http://127.15.84.62:40791/ using document root <none> and password file <none>
I20260810 14:40:29.803948 18358 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:29.804023 18358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:29.804251 18358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:29.806488 18358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "d275de245cf44508b6f355d826287a4e"
format_stamp: "Formatted at 2026-08-10 14:40:29 on dist-test-slave-rmhg"
server_key: "c7e12160518be2b0cbeebda974a97f18"
server_key_iv: "563e7e536d424fc946cad2242e685582"
server_key_version: "encryptionkey@0"
I20260810 14:40:29.807163 18358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "d275de245cf44508b6f355d826287a4e"
format_stamp: "Formatted at 2026-08-10 14:40:29 on dist-test-slave-rmhg"
server_key: "c7e12160518be2b0cbeebda974a97f18"
server_key_iv: "563e7e536d424fc946cad2242e685582"
server_key_version: "encryptionkey@0"
I20260810 14:40:29.811231 18358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:40:29.813902 18387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:29.815212 18358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:40:29.815348 18358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "d275de245cf44508b6f355d826287a4e"
format_stamp: "Formatted at 2026-08-10 14:40:29 on dist-test-slave-rmhg"
server_key: "c7e12160518be2b0cbeebda974a97f18"
server_key_iv: "563e7e536d424fc946cad2242e685582"
server_key_version: "encryptionkey@0"
I20260810 14:40:29.815460 18358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:29.836951 18358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:29.840389 18358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:29.840602 18358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:29.849167 18358 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:41501
I20260810 14:40:29.849216 18499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:41501 every 8 connection(s)
I20260810 14:40:29.850238 18358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:29.853271 18500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:29.858431 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 18358
I20260810 14:40:29.858557 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:29.858810 15696 external_mini_cluster.cc:1428] Setting key edcb0b4a7ba1c89ae1c497835e835532
I20260810 14:40:29.859808 18500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e: Bootstrap starting.
I20260810 14:40:29.862449 18500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:29.863301 18500 log.cc:824] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:29.865438 18500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e: No bootstrap required, opened a new log
Aug 10 14:40:29 dist-test-slave-rmhg krb5kdc[18339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:29.869483 18500 raft_consensus.cc:348] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d275de245cf44508b6f355d826287a4e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 41501 } }
I20260810 14:40:29.869761 18500 raft_consensus.cc:374] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:29.869819 18500 raft_consensus.cc:729] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d275de245cf44508b6f355d826287a4e, State: Initialized, Role: FOLLOWER
I20260810 14:40:29.870319 18500 consensus_queue.cc:260] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d275de245cf44508b6f355d826287a4e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 41501 } }
I20260810 14:40:29.870452 18500 raft_consensus.cc:388] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:29.870518 18500 raft_consensus.cc:482] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:29.870626 18500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:29.871814 18500 raft_consensus.cc:504] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d275de245cf44508b6f355d826287a4e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 41501 } }
I20260810 14:40:29.872233 18500 leader_election.cc:302] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d275de245cf44508b6f355d826287a4e; no voters:
I20260810 14:40:29.872627 18500 leader_election.cc:288] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:29.872852 18506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:29.873063 18506 raft_consensus.cc:686] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [term 1 LEADER]: Becoming Leader. State: Replica: d275de245cf44508b6f355d826287a4e, State: Running, Role: LEADER
I20260810 14:40:29.873436 18506 consensus_queue.cc:237] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d275de245cf44508b6f355d826287a4e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 41501 } }
I20260810 14:40:29.873937 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:29.860335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.15.84.62:41501)
0810 14:40:29.860844 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:29.860861 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:29.860899 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:40:29.861691 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:29.861715 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:29.862096 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:29.862456 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:29.863303 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:29.864362 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:29.865271 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:29.865831 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:29.865993 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:29.868661 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:29.868667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:29.868690 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:29.868725 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:29.870451 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:29.870989 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:29.870995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:29.871002 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:29.871170 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:29.871410 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:29.871414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:29.871416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:29.872023 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:29.872236 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:29.872501 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 14:40:29.872863 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":161,"threads_started":1}
I20260810 14:40:29.874141 18500 sys_catalog.cc:564] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:29.874612 18506 sys_catalog.cc:455] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d275de245cf44508b6f355d826287a4e. Latest consensus state: current_term: 1 leader_uuid: "d275de245cf44508b6f355d826287a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d275de245cf44508b6f355d826287a4e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 41501 } } }
I20260810 14:40:29.874822 18506 sys_catalog.cc:458] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:29.875048 18507 sys_catalog.cc:455] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d275de245cf44508b6f355d826287a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d275de245cf44508b6f355d826287a4e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 41501 } } }
I20260810 14:40:29.875139 18507 sys_catalog.cc:458] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:29.876194 18513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:29.879454 18513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:29.885783 18513 catalog_manager.cc:1347] Generated new cluster ID: f035b2f029f94be1aaed6e7c17e89ec2
I20260810 14:40:29.885874 18513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:29.894286 18513 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:29.895375 18513 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:29.902865 18513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e: Generated new TSK 0
I20260810 14:40:29.903661 18513 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:40:29.940001 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:41501
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:30.063342 18536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:30.063643 18536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:30.063727 18536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:30.067966 18536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:30.068045 18536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:30.068089 18536 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 14:40:30.068101 18536 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:40:30.068145 18536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:40:30.073771 18536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.84.62:41501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:30.074992 18536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:30.076108 18536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:30.086614 18543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:30.086578 18547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:30.086753 18536 server_base.cc:1034] running on GCE node
W20260810 14:40:30.086578 18544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:30.087357 18536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:30.088047 18536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:30.089233 18536 hybrid_clock.cc:648] HybridClock initialized: now 1786372830089205 us; error 40 us; skew 500 ppm
Aug 10 14:40:30 dist-test-slave-rmhg krb5kdc[18339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372830, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:30.092206 18536 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:40:30.093266 18536 webserver.cc:460] Webserver started at http://127.15.84.1:35615/ using document root <none> and password file <none>
I20260810 14:40:30.093906 18536 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:30.093986 18536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:30.094223 18536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:30.096601 18536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "88a43a63d1f64217845294110e04b55a"
format_stamp: "Formatted at 2026-08-10 14:40:30 on dist-test-slave-rmhg"
server_key: "f126a6292add069453360944165bf2b7"
server_key_iv: "30298cb8c61b7f508e3f2c34f9551cf8"
server_key_version: "encryptionkey@0"
I20260810 14:40:30.097349 18536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "88a43a63d1f64217845294110e04b55a"
format_stamp: "Formatted at 2026-08-10 14:40:30 on dist-test-slave-rmhg"
server_key: "f126a6292add069453360944165bf2b7"
server_key_iv: "30298cb8c61b7f508e3f2c34f9551cf8"
server_key_version: "encryptionkey@0"
I20260810 14:40:30.101940 18536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:40:30.104755 18559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:30.106204 18536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:40:30.106351 18536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "88a43a63d1f64217845294110e04b55a"
format_stamp: "Formatted at 2026-08-10 14:40:30 on dist-test-slave-rmhg"
server_key: "f126a6292add069453360944165bf2b7"
server_key_iv: "30298cb8c61b7f508e3f2c34f9551cf8"
server_key_version: "encryptionkey@0"
I20260810 14:40:30.106482 18536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:30.135187 18536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:30.138845 18536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:30.139068 18536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:30.140781 18536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:30.140849 18536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:30.140910 18536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:30.140931 18536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:30.151894 18536 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:34265
Aug 10 14:40:30 dist-test-slave-rmhg krb5kdc[18339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372830, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:30.153232 18536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:40:30.157349 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 18536
I20260810 14:40:30.157490 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:40:30.157804 15696 external_mini_cluster.cc:1428] Setting key db0c8c0300f72cbe791c236e3c71d89d
I20260810 14:40:30.159299 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.141793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:50533 (local address 127.15.84.62:41501)
0810 14:40:30.141955 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:30.141960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:30.142987 (+ 1027us) server_negotiation.cc:407] Connection header received
0810 14:40:30.144460 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:30.144478 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:30.144552 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:30.144656 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:30.146171 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.147027 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.148929 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.149336 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.149446 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.152935 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:30.152940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:30.152943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:30.152977 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:30.154802 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.155496 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.155501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.155503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.155616 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.156084 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.156087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.156089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.156447 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:30.156548 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:30.158991 (+ 2443us) server_negotiation.cc:299] Negotiation successful
0810 14:40:30.159151 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:40:30.154145 18745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:34265 every 8 connection(s)
I20260810 14:40:30.159538 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.142251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:41501 (local address 127.15.84.1:50533)
0810 14:40:30.142780 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:30.142838 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:30.144147 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:30.144855 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:30.144864 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:30.145390 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:30.145985 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.146002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.147170 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.147174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.148746 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.148762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.149685 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.149690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.149918 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.150621 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:30.150644 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:30.152663 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:30.154980 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:30.154987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:30.155005 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:30.155365 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:30.155762 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:30.155765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:30.155767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:30.155965 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:30.156608 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:30.156616 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:30.156984 (+ 368us) client_negotiation.cc:769] Sending connection context
0810 14:40:30.157254 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 14:40:30.157680 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":125,"threads_started":1}
I20260810 14:40:30.161404 18747 heartbeater.cc:347] Connected to a master server at 127.15.84.62:41501
I20260810 14:40:30.161589 18747 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:30.162315 18747 heartbeater.cc:511] Master 127.15.84.62:41501 requested a full tablet report, sending...
I20260810 14:40:30.164556 18414 ts_manager.cc:194] Registered new tserver with Master: 88a43a63d1f64217845294110e04b55a (127.15.84.1:34265)
I20260810 14:40:30.166086 18414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:50533
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:40:30.171589 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.163985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:53721 (local address 127.15.84.62:41501)
0810 14:40:30.164115 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:30.164120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:30.164458 (+ 338us) server_negotiation.cc:407] Connection header received
0810 14:40:30.164593 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:30.164596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:30.164649 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:30.164719 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:30.165372 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.166148 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.167260 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.167493 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.167584 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.168677 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:30.168681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:30.168684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:30.168709 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:30.170058 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.170454 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.170457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.170459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.170538 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.170797 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.170801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.170803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.170983 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:30.171069 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:30.171319 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 14:40:30.171434 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:40:30.171607 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.164055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:41501 (local address 127.15.84.1:53721)
0810 14:40:30.164346 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:30.164362 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:30.164497 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:30.164847 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:30.164850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:30.165077 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:30.165244 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.165253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.166308 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.166312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.167143 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.167154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.167599 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.167602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.167703 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.168189 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:30.168204 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:30.168542 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:30.170167 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:30.170170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:30.170172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:30.170367 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:30.170619 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:30.170622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:30.170624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:30.170715 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:30.171069 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:30.171072 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:30.171167 (+ 95us) client_negotiation.cc:769] Sending connection context
0810 14:40:30.171256 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 14:40:30.171408 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:40:30.188352 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:41501
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:30.308120 18763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:30.308427 18763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:30.308478 18763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:30.312743 18763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:30.312832 18763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:30.312876 18763 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 14:40:30.312893 18763 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:40:30.312938 18763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:40:30.318730 18763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.84.62:41501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:30.319976 18763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:30.321117 18763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:30.331851 18776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:30.331857 18780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:30.331921 18763 server_base.cc:1034] running on GCE node
W20260810 14:40:30.331851 18777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:30.332579 18763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:30.333360 18763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:30.334537 18763 hybrid_clock.cc:648] HybridClock initialized: now 1786372830334503 us; error 49 us; skew 500 ppm
Aug 10 14:40:30 dist-test-slave-rmhg krb5kdc[18339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372830, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:30.337990 18763 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:40:30.339138 18763 webserver.cc:460] Webserver started at http://127.15.84.2:35285/ using document root <none> and password file <none>
I20260810 14:40:30.339773 18763 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:30.339852 18763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:30.340071 18763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:30.342378 18763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "271a88e76c214207baeddbe270db2b73"
format_stamp: "Formatted at 2026-08-10 14:40:30 on dist-test-slave-rmhg"
server_key: "8981cc73bed6ee61ceaf405d18c2da09"
server_key_iv: "dc52bb76f416f571ca31523a920c512e"
server_key_version: "encryptionkey@0"
I20260810 14:40:30.342979 18763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "271a88e76c214207baeddbe270db2b73"
format_stamp: "Formatted at 2026-08-10 14:40:30 on dist-test-slave-rmhg"
server_key: "8981cc73bed6ee61ceaf405d18c2da09"
server_key_iv: "dc52bb76f416f571ca31523a920c512e"
server_key_version: "encryptionkey@0"
I20260810 14:40:30.347162 18763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:40:30.349951 18792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:30.351267 18763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:30.351390 18763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "271a88e76c214207baeddbe270db2b73"
format_stamp: "Formatted at 2026-08-10 14:40:30 on dist-test-slave-rmhg"
server_key: "8981cc73bed6ee61ceaf405d18c2da09"
server_key_iv: "dc52bb76f416f571ca31523a920c512e"
server_key_version: "encryptionkey@0"
I20260810 14:40:30.351507 18763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:30.363265 18763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:30.366678 18763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:30.366895 18763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:30.368471 18763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:30.368530 18763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:30.368587 18763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:30.368618 18763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:40:30 dist-test-slave-rmhg krb5kdc[18339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372830, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:30.379936 18763 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:38301
I20260810 14:40:30.381271 18763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:40:30.381232 18965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:38301 every 8 connection(s)
I20260810 14:40:30.384943 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.369371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:54807 (local address 127.15.84.62:41501)
0810 14:40:30.369489 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:30.369493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:30.370628 (+ 1135us) server_negotiation.cc:407] Connection header received
0810 14:40:30.371780 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:30.371785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:30.371851 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:30.371910 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:30.373766 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.374569 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.376478 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.376860 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.376985 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.380934 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:30.380941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:30.380944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:30.380982 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:30.382856 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.383469 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.383473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.383474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.383569 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.383926 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.383928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.383929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.384121 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:30.384198 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:30.384702 (+ 504us) server_negotiation.cc:299] Negotiation successful
0810 14:40:30.384815 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 14:40:30.385866 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.369951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:41501 (local address 127.15.84.2:54807)
0810 14:40:30.370470 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:30.370517 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:30.371471 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:30.372382 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:30.372393 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:30.372845 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:30.373604 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.373629 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.374704 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.374708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.376263 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.376278 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.377070 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.377075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.377373 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.378235 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:30.378263 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:30.380618 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:30.383022 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:30.383029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:30.383045 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:30.383341 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:30.383690 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:30.383693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:30.383695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:30.383842 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:30.384203 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:30.384208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:30.384454 (+ 246us) client_negotiation.cc:769] Sending connection context
0810 14:40:30.384626 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 14:40:30.384902 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":149,"threads_started":1}
I20260810 14:40:30.386124 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 18763
I20260810 14:40:30.386229 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:40:30.386529 15696 external_mini_cluster.cc:1428] Setting key a3abe65994fcc44be4856a7732e8f023
I20260810 14:40:30.387236 18967 heartbeater.cc:347] Connected to a master server at 127.15.84.62:41501
I20260810 14:40:30.387434 18967 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:30.388088 18967 heartbeater.cc:511] Master 127.15.84.62:41501 requested a full tablet report, sending...
I20260810 14:40:30.389309 18418 ts_manager.cc:194] Registered new tserver with Master: 271a88e76c214207baeddbe270db2b73 (127.15.84.2:38301)
I20260810 14:40:30.389937 18418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:54807
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
I20260810 14:40:30.397346 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.389481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:43947 (local address 127.15.84.62:41501)
0810 14:40:30.389587 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:30.389601 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:30.390434 (+ 833us) server_negotiation.cc:407] Connection header received
0810 14:40:30.390601 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:30.390605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:30.390656 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:30.390740 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:30.391251 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.391991 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.392806 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.393023 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.393110 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.394215 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:30.394219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:30.394221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:30.394246 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:30.395530 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.395952 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.395955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.395956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.396032 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.396295 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.396299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.396301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.396653 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:30.396747 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:30.397013 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 14:40:30.397127 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:40:30.397349 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.389410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:41501 (local address 127.15.84.2:43947)
0810 14:40:30.390338 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:30.390353 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:30.390458 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:30.390764 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:30.390767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:30.390990 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:30.391128 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.391135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.392118 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.392121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.392695 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.392704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.393132 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.393135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.393253 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.393734 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:30.393746 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:30.394075 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:30.395659 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:30.395662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:30.395664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:30.395864 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:30.396123 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:30.396126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:30.396128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:30.396213 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:30.396750 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:30.396753 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:30.396854 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 14:40:30.396946 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 14:40:30.397119 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:40:30.414304 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:41501
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:30.534897 18981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:30.535216 18981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:30.535267 18981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:30.539515 18981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:30.539597 18981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:30.539637 18981 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 14:40:30.539652 18981 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:40:30.539714 18981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:40:30.545303 18981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.84.62:41501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:30.546468 18981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:30.547518 18981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:30.557509 18992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:30.557523 18988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:30.557507 18987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:30.557772 18981 server_base.cc:1034] running on GCE node
I20260810 14:40:30.558370 18981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:30.559069 18981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:30.560226 18981 hybrid_clock.cc:648] HybridClock initialized: now 1786372830560196 us; error 50 us; skew 500 ppm
Aug 10 14:40:30 dist-test-slave-rmhg krb5kdc[18339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372830, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:30.563215 18981 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:40:30.564220 18981 webserver.cc:460] Webserver started at http://127.15.84.3:32829/ using document root <none> and password file <none>
I20260810 14:40:30.564837 18981 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:30.564911 18981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:30.565129 18981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:30.567368 18981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "e06b6bfa76674c9a9d828987d69162a8"
format_stamp: "Formatted at 2026-08-10 14:40:30 on dist-test-slave-rmhg"
server_key: "50ce7cf36873081feaad27662bd2e1c3"
server_key_iv: "041c87f3a1ef69b75a11a8f659af9177"
server_key_version: "encryptionkey@0"
I20260810 14:40:30.567965 18981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "e06b6bfa76674c9a9d828987d69162a8"
format_stamp: "Formatted at 2026-08-10 14:40:30 on dist-test-slave-rmhg"
server_key: "50ce7cf36873081feaad27662bd2e1c3"
server_key_iv: "041c87f3a1ef69b75a11a8f659af9177"
server_key_version: "encryptionkey@0"
I20260810 14:40:30.572311 18981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:40:30.575325 18999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:30.576817 18981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:40:30.576967 18981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "e06b6bfa76674c9a9d828987d69162a8"
format_stamp: "Formatted at 2026-08-10 14:40:30 on dist-test-slave-rmhg"
server_key: "50ce7cf36873081feaad27662bd2e1c3"
server_key_iv: "041c87f3a1ef69b75a11a8f659af9177"
server_key_version: "encryptionkey@0"
I20260810 14:40:30.577099 18981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:30.587343 18981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:30.590904 18981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:30.591074 18981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:30.592638 18981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:30.592691 18981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:30.592734 18981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:30.592764 18981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:40:30 dist-test-slave-rmhg krb5kdc[18339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372830, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:30.602939 18981 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:34533
I20260810 14:40:30.602993 19179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:34533 every 8 connection(s)
I20260810 14:40:30.604019 18981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:40:30.607076 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.593478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:54123 (local address 127.15.84.62:41501)
0810 14:40:30.593598 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:30.593601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:30.594792 (+ 1191us) server_negotiation.cc:407] Connection header received
0810 14:40:30.595861 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:30.595865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:30.595929 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:30.595990 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:30.597821 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.598614 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.600283 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.600514 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.600610 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.603643 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:30.603646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:30.603649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:30.603685 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:30.605345 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.605820 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.605823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.605824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.605895 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.606178 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.606180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.606181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.606352 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:30.606422 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:30.606853 (+ 431us) server_negotiation.cc:299] Negotiation successful
0810 14:40:30.606953 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 14:40:30.607965 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.594095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:41501 (local address 127.15.84.3:54123)
0810 14:40:30.594617 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:30.594680 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:30.595583 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:30.596472 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:30.596480 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:30.596898 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:30.597657 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.597680 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.598748 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.598752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.600145 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.600157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.600668 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.600672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.600895 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.601627 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:30.601652 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:30.603451 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:30.605446 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:30.605450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:30.605462 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:30.605737 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:30.605963 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:30.605965 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:30.605966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:30.606113 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:30.606422 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:30.606427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:30.606670 (+ 243us) client_negotiation.cc:769] Sending connection context
0810 14:40:30.606825 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 14:40:30.607099 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":145,"threads_started":1}
I20260810 14:40:30.609189 19180 heartbeater.cc:347] Connected to a master server at 127.15.84.62:41501
I20260810 14:40:30.609338 19180 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:30.609905 19180 heartbeater.cc:511] Master 127.15.84.62:41501 requested a full tablet report, sending...
I20260810 14:40:30.611106 18418 ts_manager.cc:194] Registered new tserver with Master: e06b6bfa76674c9a9d828987d69162a8 (127.15.84.3:34533)
I20260810 14:40:30.611886 18418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:54123
I20260810 14:40:30.612612 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 18981
I20260810 14:40:30.612739 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:40:30.613035 15696 external_mini_cluster.cc:1428] Setting key 7ae456d942592235c0870d4c01f8cbe9
I20260810 14:40:30.615031 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:40:30.619361 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.611666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:47563 (local address 127.15.84.62:41501)
0810 14:40:30.611813 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:30.611817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:30.612074 (+ 257us) server_negotiation.cc:407] Connection header received
0810 14:40:30.612240 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:30.612244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:30.612286 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:30.612359 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:30.612885 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.613664 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.614749 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.614976 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.615068 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.616201 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:30.616205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:30.616207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:30.616233 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:30.617711 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.618143 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.618147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.618149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.618244 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.618501 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.618504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.618505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.618754 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:30.618857 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:30.619119 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 14:40:30.619224 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:40:30.619405 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.611561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:41501 (local address 127.15.84.3:47563)
0810 14:40:30.611986 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:30.612001 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:30.612118 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:30.612388 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:30.612391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:30.612622 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:30.612745 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.612752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.613830 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.613834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.614621 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.614632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.615091 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.615096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.615192 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.615701 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:30.615713 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:30.616048 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:30.617823 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:30.617826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:30.617828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:30.618053 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:30.618327 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:30.618329 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:30.618331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:30.618418 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:30.618846 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:30.618849 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:30.618958 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 14:40:30.619061 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 14:40:30.619225 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":134,"threads_started":1}
Aug 10 14:40:30 dist-test-slave-rmhg krb5kdc[18339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:40:30 dist-test-slave-rmhg krb5kdc[18339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:30.643900 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.635582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33014 (local address 127.15.84.62:41501)
0810 14:40:30.635750 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:30.635754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:30.635770 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:40:30.635818 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:30.635821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:30.635863 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:30.635954 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:30.636706 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.637324 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.638132 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.638357 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.638433 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.640729 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:30.640732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:30.640734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:30.640765 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:30.642229 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.642700 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.642704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.642705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.642796 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.643077 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.643080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.643081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.643291 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:30.643398 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:30.643662 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 14:40:30.643763 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 14:40:30.647121 18418 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:40:30.650228 18418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:30.665709 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.660280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.15.84.2:38301)
0810 14:40:30.660721 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:30.660733 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:30.661005 (+ 272us) server_negotiation.cc:407] Connection header received
0810 14:40:30.661104 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:30.661110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:30.661340 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:30.661766 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:30.661911 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.663014 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.664247 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.664937 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.665045 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.665258 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:30.665445 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 14:40:30.665542 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":131,"threads_started":1}
I20260810 14:40:30.665697 19208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.659442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:38301 (local address 127.0.0.1:42322)
0810 14:40:30.660810 (+ 1368us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:30.660824 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:30.660911 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:30.661504 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:30.661507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:30.661526 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:30.661622 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.661629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.663122 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.663126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.664103 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.664113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.665076 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.665079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.665241 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.665289 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 14:40:30.665398 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 14:40:30.665535 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"mutex_wait_us":3,"thread_start_us":112,"threads_started":1}
I20260810 14:40:30.665885 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.659399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:34265 (local address 127.0.0.1:41938)
0810 14:40:30.660071 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:30.660149 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:30.660297 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:30.660943 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:30.660951 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:30.660973 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:30.661142 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.661184 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.662854 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.662857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.664324 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.664331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.665495 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.665499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.665594 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.665616 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:40:30.665668 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 14:40:30.665744 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":91,"thread_start_us":100,"threads_started":1}
I20260810 14:40:30.666293 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.659656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:34533 (local address 127.0.0.1:50884)
0810 14:40:30.660914 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:30.660927 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:30.661024 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:30.661630 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:30.661633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:30.661649 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:30.661761 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.661767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.662807 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.662810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.664316 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.664328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.665704 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.665707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.665785 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.665805 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 14:40:30.666090 (+ 285us) client_negotiation.cc:240] Negotiation successful
0810 14:40:30.666170 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"mutex_wait_us":64,"thread_start_us":104,"threads_started":1}
I20260810 14:40:30.666504 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.660280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.15.84.3:34533)
0810 14:40:30.660672 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:30.660684 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:30.661005 (+ 321us) server_negotiation.cc:407] Connection header received
0810 14:40:30.661136 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:30.661141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:30.661365 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:30.661559 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:30.661869 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.662704 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.664482 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.665355 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.665460 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.665634 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:30.666247 (+ 613us) server_negotiation.cc:299] Negotiation successful
0810 14:40:30.666363 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":122,"threads_started":1}
I20260810 14:40:30.667457 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.659572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.15.84.1:34265)
0810 14:40:30.660366 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:30.660381 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:30.660406 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:40:30.660483 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:30.660488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:30.660728 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:30.661337 (+ 609us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:30.661493 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.662714 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.664484 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.665365 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.666222 (+ 857us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.666380 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:30.666434 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 14:40:30.666527 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":84,"threads_started":1}
I20260810 14:40:30.669031 18864 tablet_service.cc:1467] Processing CreateTablet for tablet a41757781d404c74b5f09bad58f9fa03 (DEFAULT_TABLE table=test-table [id=2fa4ec56b2454ea48a2f54b4a9429241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:30.669333 19077 tablet_service.cc:1467] Processing CreateTablet for tablet a41757781d404c74b5f09bad58f9fa03 (DEFAULT_TABLE table=test-table [id=2fa4ec56b2454ea48a2f54b4a9429241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:30.670137 18864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41757781d404c74b5f09bad58f9fa03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:30.670436 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41757781d404c74b5f09bad58f9fa03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:30.670213 18647 tablet_service.cc:1467] Processing CreateTablet for tablet a41757781d404c74b5f09bad58f9fa03 (DEFAULT_TABLE table=test-table [id=2fa4ec56b2454ea48a2f54b4a9429241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:30.671216 18647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41757781d404c74b5f09bad58f9fa03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:30.677431 19211 tablet_bootstrap.cc:492] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73: Bootstrap starting.
I20260810 14:40:30.679390 19212 tablet_bootstrap.cc:492] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8: Bootstrap starting.
I20260810 14:40:30.680399 19211 tablet_bootstrap.cc:654] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:30.681170 19211 log.cc:824] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:30.681263 19212 tablet_bootstrap.cc:654] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:30.682056 19212 log.cc:824] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:30.682744 19215 tablet_bootstrap.cc:492] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a: Bootstrap starting.
I20260810 14:40:30.683218 19211 tablet_bootstrap.cc:492] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73: No bootstrap required, opened a new log
I20260810 14:40:30.683408 19211 ts_tablet_manager.cc:1397] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:40:30.684633 19215 tablet_bootstrap.cc:654] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:30.685387 19215 log.cc:824] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:30.686818 19211 raft_consensus.cc:348] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:30.687049 19211 raft_consensus.cc:374] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:30.687085 19211 raft_consensus.cc:729] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271a88e76c214207baeddbe270db2b73, State: Initialized, Role: FOLLOWER
I20260810 14:40:30.687513 19211 consensus_queue.cc:260] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:30.688148 19212 tablet_bootstrap.cc:492] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8: No bootstrap required, opened a new log
I20260810 14:40:30.688197 18967 heartbeater.cc:503] Master 127.15.84.62:41501 was elected leader, sending a full tablet report...
I20260810 14:40:30.688349 19212 ts_tablet_manager.cc:1397] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260810 14:40:30.688474 19211 ts_tablet_manager.cc:1428] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260810 14:40:30.687633 19215 tablet_bootstrap.cc:492] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a: No bootstrap required, opened a new log
I20260810 14:40:30.691560 19215 ts_tablet_manager.cc:1397] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260810 14:40:30.691541 19212 raft_consensus.cc:348] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:30.691829 19212 raft_consensus.cc:374] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:30.691890 19212 raft_consensus.cc:729] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06b6bfa76674c9a9d828987d69162a8, State: Initialized, Role: FOLLOWER
I20260810 14:40:30.692375 19212 consensus_queue.cc:260] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:30.693231 19180 heartbeater.cc:503] Master 127.15.84.62:41501 was elected leader, sending a full tablet report...
I20260810 14:40:30.693575 19212 ts_tablet_manager.cc:1428] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260810 14:40:30.695917 19215 raft_consensus.cc:348] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:30.696233 19215 raft_consensus.cc:374] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:30.696301 19215 raft_consensus.cc:729] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a43a63d1f64217845294110e04b55a, State: Initialized, Role: FOLLOWER
I20260810 14:40:30.696835 19215 consensus_queue.cc:260] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:30.697577 18747 heartbeater.cc:503] Master 127.15.84.62:41501 was elected leader, sending a full tablet report...
I20260810 14:40:30.697911 19215 ts_tablet_manager.cc:1428] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
W20260810 14:40:30.855232 19182 tablet.cc:2369] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:40:30.881193 19223 raft_consensus.cc:482] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:30.881449 19223 raft_consensus.cc:504] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:30.882761 19223 leader_election.cc:288] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 271a88e76c214207baeddbe270db2b73 (127.15.84.2:38301), e06b6bfa76674c9a9d828987d69162a8 (127.15.84.3:34533)
W20260810 14:40:30.883333 18968 tablet.cc:2369] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:40:30.887101 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.883235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:49443 (local address 127.15.84.2:38301)
0810 14:40:30.883380 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:30.883384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:30.883847 (+ 463us) server_negotiation.cc:407] Connection header received
0810 14:40:30.884041 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:30.884044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:30.884091 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:30.884153 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:30.884480 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.884992 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.886132 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.886541 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.886648 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.886719 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:30.886883 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 14:40:30.886960 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:40:30.887188 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.883259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:52531 (local address 127.15.84.3:34533)
0810 14:40:30.883380 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:30.883384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:30.883804 (+ 420us) server_negotiation.cc:407] Connection header received
0810 14:40:30.883971 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:30.883974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:30.884030 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:30.884109 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:30.884427 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.884971 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.886117 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.886623 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.886731 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.886795 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:30.886966 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 14:40:30.887039 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:40:30.887239 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.883075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:38301 (local address 127.15.84.1:49443)
0810 14:40:30.883766 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:30.883779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:30.883904 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:30.884183 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:30.884185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:30.884228 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:30.884357 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.884363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.885102 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.885105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.886001 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.886011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.886661 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.886664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.886762 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.886789 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 14:40:30.886859 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 14:40:30.886953 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":354,"spinlock_wait_cycles":768,"thread_start_us":113,"threads_started":1}
I20260810 14:40:30.887265 19224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.883073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:34533 (local address 127.15.84.1:52531)
0810 14:40:30.883682 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:30.883699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:30.883849 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:30.884141 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:30.884145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:30.884164 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:30.884319 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.884324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.885085 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.885088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.885990 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.886001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.886740 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.886743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.886852 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.886875 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:40:30.886943 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 14:40:30.887017 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":8,"spinlock_wait_cycles":10624,"thread_start_us":137,"threads_started":1}
I20260810 14:40:30.887898 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41757781d404c74b5f09bad58f9fa03" candidate_uuid: "88a43a63d1f64217845294110e04b55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e06b6bfa76674c9a9d828987d69162a8" is_pre_election: true
I20260810 14:40:30.887938 18890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41757781d404c74b5f09bad58f9fa03" candidate_uuid: "88a43a63d1f64217845294110e04b55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271a88e76c214207baeddbe270db2b73" is_pre_election: true
I20260810 14:40:30.888263 19106 raft_consensus.cc:2454] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a43a63d1f64217845294110e04b55a in term 0.
I20260810 14:40:30.888263 18890 raft_consensus.cc:2454] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a43a63d1f64217845294110e04b55a in term 0.
I20260810 14:40:30.888785 18565 leader_election.cc:302] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a43a63d1f64217845294110e04b55a, e06b6bfa76674c9a9d828987d69162a8; no voters:
I20260810 14:40:30.889063 19223 raft_consensus.cc:2781] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:40:30.889194 19223 raft_consensus.cc:482] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:40:30.889253 19223 raft_consensus.cc:3037] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:30.890347 19223 raft_consensus.cc:504] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:30.890725 19223 leader_election.cc:288] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [CANDIDATE]: Term 1 election: Requested vote from peers 271a88e76c214207baeddbe270db2b73 (127.15.84.2:38301), e06b6bfa76674c9a9d828987d69162a8 (127.15.84.3:34533)
I20260810 14:40:30.891122 18890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41757781d404c74b5f09bad58f9fa03" candidate_uuid: "88a43a63d1f64217845294110e04b55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271a88e76c214207baeddbe270db2b73"
I20260810 14:40:30.891127 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41757781d404c74b5f09bad58f9fa03" candidate_uuid: "88a43a63d1f64217845294110e04b55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e06b6bfa76674c9a9d828987d69162a8"
I20260810 14:40:30.891260 19106 raft_consensus.cc:3037] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:30.891254 18890 raft_consensus.cc:3037] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:30.892390 18890 raft_consensus.cc:2454] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a43a63d1f64217845294110e04b55a in term 1.
I20260810 14:40:30.892786 18564 leader_election.cc:302] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 271a88e76c214207baeddbe270db2b73, 88a43a63d1f64217845294110e04b55a; no voters:
I20260810 14:40:30.892792 19106 raft_consensus.cc:2454] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a43a63d1f64217845294110e04b55a in term 1.
I20260810 14:40:30.892997 19223 raft_consensus.cc:2781] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:30.893292 19223 raft_consensus.cc:686] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [term 1 LEADER]: Becoming Leader. State: Replica: 88a43a63d1f64217845294110e04b55a, State: Running, Role: LEADER
I20260810 14:40:30.893668 19223 consensus_queue.cc:237] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:30.897397 18414 catalog_manager.cc:5495] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a43a63d1f64217845294110e04b55a (127.15.84.1). New cstate: current_term: 1 leader_uuid: "88a43a63d1f64217845294110e04b55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } health_report { overall_health: UNKNOWN } } }
W20260810 14:40:30.905488 18748 tablet.cc:2369] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 14:40:30 dist-test-slave-rmhg krb5kdc[18339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372829, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:30.987360 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.979097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.15.84.62:41501)
0810 14:40:30.979246 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:30.979251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:30.979432 (+ 181us) server_negotiation.cc:407] Connection header received
0810 14:40:30.979563 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:30.979566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:30.979605 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:30.979679 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:30.980406 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.980976 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.981866 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.982076 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.982164 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.984436 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:30.984439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:30.984441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:30.984463 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:30.985660 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.986119 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.986122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.986123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.986194 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:30.986445 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:30.986448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:30.986450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:30.986627 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:30.986721 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:30.987002 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 14:40:30.987161 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":58}
I20260810 14:40:30.987360 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:30.979047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:41501 (local address 127.0.0.1:33030)
0810 14:40:30.979340 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:30.979353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:30.979479 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:30.979705 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:30.979708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:30.980160 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:30.980301 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.980308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.981079 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.981082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.981747 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:30.981758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:30.982185 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:30.982189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:30.982262 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:30.982798 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:30.982810 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:30.984298 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:30.985783 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:30.985795 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:30.985803 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:30.986030 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:30.986271 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:30.986274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:30.986276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:30.986367 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:30.986722 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:30.986730 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:30.986846 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 14:40:30.986943 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 14:40:30.987106 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":111,"threads_started":1}
I20260810 14:40:30.991695 18414 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:33030:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 14:40:30.992455 18414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:30.997298 19077 tablet_service.cc:1467] Processing CreateTablet for tablet ab0de0c7b3de487bb437f994afc49687 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cd9a354da62941d093caa7a60061e952]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:40:30.997283 18647 tablet_service.cc:1467] Processing CreateTablet for tablet ab0de0c7b3de487bb437f994afc49687 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cd9a354da62941d093caa7a60061e952]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:40:30.997278 18864 tablet_service.cc:1467] Processing CreateTablet for tablet ab0de0c7b3de487bb437f994afc49687 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cd9a354da62941d093caa7a60061e952]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:40:30.998065 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0de0c7b3de487bb437f994afc49687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:30.998075 18647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0de0c7b3de487bb437f994afc49687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:30.998065 18864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0de0c7b3de487bb437f994afc49687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:31.000772 19212 tablet_bootstrap.cc:492] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8: Bootstrap starting.
I20260810 14:40:31.001138 19215 tablet_bootstrap.cc:492] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a: Bootstrap starting.
I20260810 14:40:31.001874 19212 tablet_bootstrap.cc:654] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:31.002399 19215 tablet_bootstrap.cc:654] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:31.002640 19211 tablet_bootstrap.cc:492] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73: Bootstrap starting.
I20260810 14:40:31.003328 19212 tablet_bootstrap.cc:492] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8: No bootstrap required, opened a new log
I20260810 14:40:31.003432 19212 ts_tablet_manager.cc:1397] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:40:31.003628 19211 tablet_bootstrap.cc:654] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:31.003661 19215 tablet_bootstrap.cc:492] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a: No bootstrap required, opened a new log
I20260810 14:40:31.003747 19215 ts_tablet_manager.cc:1397] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 14:40:31.003973 19212 raft_consensus.cc:348] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.004109 19212 raft_consensus.cc:374] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:31.004154 19212 raft_consensus.cc:729] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06b6bfa76674c9a9d828987d69162a8, State: Initialized, Role: FOLLOWER
I20260810 14:40:31.004159 19215 raft_consensus.cc:348] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.004292 19215 raft_consensus.cc:374] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:31.004328 19215 raft_consensus.cc:729] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a43a63d1f64217845294110e04b55a, State: Initialized, Role: FOLLOWER
I20260810 14:40:31.004456 19215 consensus_queue.cc:260] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.004741 19223 tablet_replica.cc:439] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } } }
I20260810 14:40:31.004900 19223 tablet_replica.cc:442] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:40:31.005111 19211 tablet_bootstrap.cc:492] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73: No bootstrap required, opened a new log
I20260810 14:40:31.005174 19215 ts_tablet_manager.cc:1428] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:40:31.005232 19211 ts_tablet_manager.cc:1397] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260810 14:40:31.005405 19212 consensus_queue.cc:260] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.005693 19212 ts_tablet_manager.cc:1428] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:40:31.005676 19211 raft_consensus.cc:348] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.005785 19211 raft_consensus.cc:374] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:31.005820 19211 raft_consensus.cc:729] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271a88e76c214207baeddbe270db2b73, State: Initialized, Role: FOLLOWER
I20260810 14:40:31.005939 19211 consensus_queue.cc:260] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.005700 19221 tablet_replica.cc:439] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } } }
I20260810 14:40:31.006193 19221 tablet_replica.cc:442] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:40:31.006209 19211 ts_tablet_manager.cc:1428] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 14:40:31.006417 19220 tablet_replica.cc:439] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } } }
I20260810 14:40:31.006560 19220 tablet_replica.cc:442] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:40:31.042058 19223 raft_consensus.cc:482] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:31.042225 19223 raft_consensus.cc:504] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.042678 19223 leader_election.cc:288] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 271a88e76c214207baeddbe270db2b73 (127.15.84.2:38301), e06b6bfa76674c9a9d828987d69162a8 (127.15.84.3:34533)
I20260810 14:40:31.043156 18890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0de0c7b3de487bb437f994afc49687" candidate_uuid: "88a43a63d1f64217845294110e04b55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271a88e76c214207baeddbe270db2b73" is_pre_election: true
I20260810 14:40:31.043156 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0de0c7b3de487bb437f994afc49687" candidate_uuid: "88a43a63d1f64217845294110e04b55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e06b6bfa76674c9a9d828987d69162a8" is_pre_election: true
I20260810 14:40:31.043326 19106 raft_consensus.cc:2454] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a43a63d1f64217845294110e04b55a in term 0.
I20260810 14:40:31.043326 18890 raft_consensus.cc:2454] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a43a63d1f64217845294110e04b55a in term 0.
I20260810 14:40:31.043638 18565 leader_election.cc:302] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 271a88e76c214207baeddbe270db2b73, 88a43a63d1f64217845294110e04b55a, e06b6bfa76674c9a9d828987d69162a8; no voters:
I20260810 14:40:31.043867 19223 raft_consensus.cc:2781] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:40:31.043932 19223 raft_consensus.cc:482] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:40:31.043959 19223 raft_consensus.cc:3037] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:31.045058 19223 raft_consensus.cc:504] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.045433 19223 leader_election.cc:288] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [CANDIDATE]: Term 1 election: Requested vote from peers 271a88e76c214207baeddbe270db2b73 (127.15.84.2:38301), e06b6bfa76674c9a9d828987d69162a8 (127.15.84.3:34533)
I20260810 14:40:31.045885 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0de0c7b3de487bb437f994afc49687" candidate_uuid: "88a43a63d1f64217845294110e04b55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e06b6bfa76674c9a9d828987d69162a8"
I20260810 14:40:31.045904 18890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0de0c7b3de487bb437f994afc49687" candidate_uuid: "88a43a63d1f64217845294110e04b55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271a88e76c214207baeddbe270db2b73"
I20260810 14:40:31.046020 19106 raft_consensus.cc:3037] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:31.046031 18890 raft_consensus.cc:3037] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:31.046969 19106 raft_consensus.cc:2454] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a43a63d1f64217845294110e04b55a in term 1.
I20260810 14:40:31.046975 18890 raft_consensus.cc:2454] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a43a63d1f64217845294110e04b55a in term 1.
I20260810 14:40:31.047293 18564 leader_election.cc:302] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 271a88e76c214207baeddbe270db2b73, 88a43a63d1f64217845294110e04b55a, e06b6bfa76674c9a9d828987d69162a8; no voters:
I20260810 14:40:31.047485 19223 raft_consensus.cc:2781] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:31.047588 19223 raft_consensus.cc:686] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [term 1 LEADER]: Becoming Leader. State: Replica: 88a43a63d1f64217845294110e04b55a, State: Running, Role: LEADER
I20260810 14:40:31.047741 19223 consensus_queue.cc:237] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.048477 19227 tablet_replica.cc:439] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a: TxnStatusTablet state changed. Reason: New leader 88a43a63d1f64217845294110e04b55a. Latest consensus state: current_term: 1 leader_uuid: "88a43a63d1f64217845294110e04b55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } } }
I20260810 14:40:31.048645 19227 tablet_replica.cc:442] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:40:31.048985 19240 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 14:40:31.049247 19240 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 14:40:31.049600 18414 catalog_manager.cc:5495] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a43a63d1f64217845294110e04b55a (127.15.84.1). New cstate: current_term: 1 leader_uuid: "88a43a63d1f64217845294110e04b55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } health_report { overall_health: UNKNOWN } } }
I20260810 14:40:31.063097 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:31.059020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.15.84.1:34265)
0810 14:40:31.059183 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:31.059188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:31.059205 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:40:31.059334 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:31.059337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:31.059411 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:31.059523 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:31.059747 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.060568 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.061521 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.061761 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.061857 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:31.062180 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:31.062688 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:31.062793 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:31.062876 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 14:40:31.062944 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:40:31.063097 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:31.058906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:34265 (local address 127.0.0.1:41948)
0810 14:40:31.059068 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:31.059085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:31.059191 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:31.059528 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:31.059530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:31.059539 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:40:31.059644 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:31.059648 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.060667 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.060669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:31.061383 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:31.061415 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.061882 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.061886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:31.061974 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:31.062025 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:40:31.062780 (+ 755us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:40:31.062799 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 14:40:31.062866 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 14:40:31.062958 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 14:40:31.068976 19106 raft_consensus.cc:1261] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Refusing update from remote peer 88a43a63d1f64217845294110e04b55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:31.069005 18890 raft_consensus.cc:1261] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Refusing update from remote peer 88a43a63d1f64217845294110e04b55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:31.069785 19227 consensus_queue.cc:1035] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:31.070004 19223 consensus_queue.cc:1035] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:31.074506 19221 tablet_replica.cc:439] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8: TxnStatusTablet state changed. Reason: New leader 88a43a63d1f64217845294110e04b55a. Latest consensus state: current_term: 1 leader_uuid: "88a43a63d1f64217845294110e04b55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } } }
I20260810 14:40:31.074800 19221 tablet_replica.cc:442] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:40:31.074894 19220 tablet_replica.cc:439] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73: TxnStatusTablet state changed. Reason: New leader 88a43a63d1f64217845294110e04b55a. Latest consensus state: current_term: 1 leader_uuid: "88a43a63d1f64217845294110e04b55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } } }
I20260810 14:40:31.075033 19220 tablet_replica.cc:442] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:40:31.079712 19227 tablet_replica.cc:439] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88a43a63d1f64217845294110e04b55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } } }
I20260810 14:40:31.079856 19227 tablet_replica.cc:442] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:40:31.079967 19227 tablet_replica.cc:439] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88a43a63d1f64217845294110e04b55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } } }
I20260810 14:40:31.080026 19227 tablet_replica.cc:442] T ab0de0c7b3de487bb437f994afc49687 P 88a43a63d1f64217845294110e04b55a: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:40:31.081229 19220 tablet_replica.cc:439] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88a43a63d1f64217845294110e04b55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } } }
I20260810 14:40:31.081364 19220 tablet_replica.cc:442] T ab0de0c7b3de487bb437f994afc49687 P 271a88e76c214207baeddbe270db2b73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:40:31.084839 19221 tablet_replica.cc:439] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88a43a63d1f64217845294110e04b55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } } }
I20260810 14:40:31.084981 19221 tablet_replica.cc:442] T ab0de0c7b3de487bb437f994afc49687 P e06b6bfa76674c9a9d828987d69162a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:40:31.086524 19249 mvcc.cc:205] Tried to move back new op lower bound from 7316983116051021824 to 7316983115972968448. Current Snapshot: MvccSnapshot[applied={T|T < 7316983116051021824 or (T in {7316983116051021824})}]
I20260810 14:40:31.091321 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:31.088152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.15.84.1:34265)
0810 14:40:31.088286 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:31.088290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:31.088366 (+ 76us) server_negotiation.cc:407] Connection header received
0810 14:40:31.088515 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:31.088518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:31.088566 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:31.088627 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:31.088892 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.089468 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.090325 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.090537 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.090621 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:31.090886 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:31.090963 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:31.091022 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:31.091113 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 14:40:31.091167 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 14:40:31.103188 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:31.099895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:33819 (local address 127.15.84.1:34265)
0810 14:40:31.100012 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:31.100017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:31.100219 (+ 202us) server_negotiation.cc:407] Connection header received
0810 14:40:31.100347 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:31.100350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:31.100393 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:31.100453 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:31.100753 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.101305 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.102133 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.102348 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.102431 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:31.102757 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:31.102835 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:31.102894 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:31.102980 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 14:40:31.103024 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 14:40:31.103204 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:31.099842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:34265 (local address 127.15.84.1:33819)
0810 14:40:31.100133 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:31.100144 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:31.100262 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:31.100487 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:31.100490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:31.100497 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:40:31.100646 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:31.100652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.101400 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.101403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:31.102029 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:31.102037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.102460 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.102464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:31.102561 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:31.102608 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:40:31.102905 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:40:31.102910 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:40:31.102964 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 14:40:31.103046 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":111,"threads_started":1}
I20260810 14:40:31.109767 19106 raft_consensus.cc:1261] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Refusing update from remote peer 88a43a63d1f64217845294110e04b55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:31.109823 18890 raft_consensus.cc:1261] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Refusing update from remote peer 88a43a63d1f64217845294110e04b55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:31.110230 19227 consensus_queue.cc:1035] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:31.110460 19253 consensus_queue.cc:1035] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:31.114114 19245 mvcc.cc:205] Tried to move back new op lower bound from 7316983116222210048 to 7316983115343384576. Current Snapshot: MvccSnapshot[applied={T|T < 7316983116222210048}]
I20260810 14:40:31.155092 18418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:31.155319 18418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:31.158345 18418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e: Sending DeleteTablet for 3 replicas of tablet a41757781d404c74b5f09bad58f9fa03
I20260810 14:40:31.159353 18647 tablet_service.cc:1514] Processing DeleteTablet for tablet a41757781d404c74b5f09bad58f9fa03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:31 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:41938
I20260810 14:40:31.159497 18864 tablet_service.cc:1514] Processing DeleteTablet for tablet a41757781d404c74b5f09bad58f9fa03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:31 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:42322
I20260810 14:40:31.159930 19077 tablet_service.cc:1514] Processing DeleteTablet for tablet a41757781d404c74b5f09bad58f9fa03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:31 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:50884
I20260810 14:40:31.160445 19274 tablet_replica.cc:330] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8: stopping tablet replica
I20260810 14:40:31.160883 19274 raft_consensus.cc:2229] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:31.161226 19274 raft_consensus.cc:2258] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:31.161610 18418 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:40:31.162138 18418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:31.163172 19274 ts_tablet_manager.cc:1905] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:31.166534 19274 ts_tablet_manager.cc:1918] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:40:31.166746 19274 log.cc:1196] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/a41757781d404c74b5f09bad58f9fa03
I20260810 14:40:31.167268 19274 ts_tablet_manager.cc:1939] T a41757781d404c74b5f09bad58f9fa03 P e06b6bfa76674c9a9d828987d69162a8: Deleting consensus metadata
I20260810 14:40:31.167867 18647 tablet_service.cc:1467] Processing CreateTablet for tablet b387c453397d48f2a75232378e631af8 (DEFAULT_TABLE table=test-table [id=aaf8ee343da84f51b06928cd1d1556d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:31.168669 18864 tablet_service.cc:1467] Processing CreateTablet for tablet b387c453397d48f2a75232378e631af8 (DEFAULT_TABLE table=test-table [id=aaf8ee343da84f51b06928cd1d1556d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:31.168706 19077 tablet_service.cc:1467] Processing CreateTablet for tablet b387c453397d48f2a75232378e631af8 (DEFAULT_TABLE table=test-table [id=aaf8ee343da84f51b06928cd1d1556d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:31.168809 18392 catalog_manager.cc:4841] TS e06b6bfa76674c9a9d828987d69162a8 (127.15.84.3:34533): tablet a41757781d404c74b5f09bad58f9fa03 (table test-table [id=2fa4ec56b2454ea48a2f54b4a9429241]) successfully deleted
I20260810 14:40:31.168978 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b387c453397d48f2a75232378e631af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:31.168991 18864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b387c453397d48f2a75232378e631af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:31.169353 19270 tablet_replica.cc:330] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73: stopping tablet replica
I20260810 14:40:31.169721 19270 raft_consensus.cc:2229] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:31.169957 19270 raft_consensus.cc:2258] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:31.171132 19211 tablet_bootstrap.cc:492] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73: Bootstrap starting.
I20260810 14:40:31.171504 19212 tablet_bootstrap.cc:492] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8: Bootstrap starting.
I20260810 14:40:31.172266 19211 tablet_bootstrap.cc:654] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:31.172448 19212 tablet_bootstrap.cc:654] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:31.172490 18647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b387c453397d48f2a75232378e631af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:31.172977 19269 tablet_replica.cc:330] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a: stopping tablet replica
I20260810 14:40:31.173390 19269 raft_consensus.cc:2229] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:40:31.173830 19269 raft_consensus.cc:2258] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:31.174722 19212 tablet_bootstrap.cc:492] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8: No bootstrap required, opened a new log
I20260810 14:40:31.174819 19212 ts_tablet_manager.cc:1397] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:40:31.175308 19212 raft_consensus.cc:348] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.175401 19215 tablet_bootstrap.cc:492] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a: Bootstrap starting.
I20260810 14:40:31.175422 19212 raft_consensus.cc:374] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:31.175454 19212 raft_consensus.cc:729] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06b6bfa76674c9a9d828987d69162a8, State: Initialized, Role: FOLLOWER
I20260810 14:40:31.175576 19212 consensus_queue.cc:260] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.175637 19269 ts_tablet_manager.cc:1905] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:31.175836 19212 ts_tablet_manager.cc:1428] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:31.176465 19215 tablet_bootstrap.cc:654] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:31.177479 19270 ts_tablet_manager.cc:1905] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:31.180547 19270 ts_tablet_manager.cc:1918] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:40:31.180588 19269 ts_tablet_manager.cc:1918] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:40:31.180675 19270 log.cc:1196] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/a41757781d404c74b5f09bad58f9fa03
I20260810 14:40:31.180698 19269 log.cc:1196] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/wal/wals/a41757781d404c74b5f09bad58f9fa03
I20260810 14:40:31.180951 19211 tablet_bootstrap.cc:492] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73: No bootstrap required, opened a new log
I20260810 14:40:31.181006 19270 ts_tablet_manager.cc:1939] T a41757781d404c74b5f09bad58f9fa03 P 271a88e76c214207baeddbe270db2b73: Deleting consensus metadata
I20260810 14:40:31.181035 19269 ts_tablet_manager.cc:1939] T a41757781d404c74b5f09bad58f9fa03 P 88a43a63d1f64217845294110e04b55a: Deleting consensus metadata
I20260810 14:40:31.181046 19211 ts_tablet_manager.cc:1397] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260810 14:40:31.181631 19211 raft_consensus.cc:348] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.181763 19211 raft_consensus.cc:374] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:31.181797 19211 raft_consensus.cc:729] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271a88e76c214207baeddbe270db2b73, State: Initialized, Role: FOLLOWER
I20260810 14:40:31.181932 19211 consensus_queue.cc:260] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.182224 19211 ts_tablet_manager.cc:1428] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:31.182348 18390 catalog_manager.cc:4841] TS 271a88e76c214207baeddbe270db2b73 (127.15.84.2:38301): tablet a41757781d404c74b5f09bad58f9fa03 (table test-table [id=2fa4ec56b2454ea48a2f54b4a9429241]) successfully deleted
I20260810 14:40:31.182725 18389 catalog_manager.cc:4841] TS 88a43a63d1f64217845294110e04b55a (127.15.84.1:34265): tablet a41757781d404c74b5f09bad58f9fa03 (table test-table [id=2fa4ec56b2454ea48a2f54b4a9429241]) successfully deleted
I20260810 14:40:31.182958 19215 tablet_bootstrap.cc:492] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a: No bootstrap required, opened a new log
I20260810 14:40:31.183056 19215 ts_tablet_manager.cc:1397] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260810 14:40:31.183490 19215 raft_consensus.cc:348] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.183609 19215 raft_consensus.cc:374] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:31.183646 19215 raft_consensus.cc:729] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a43a63d1f64217845294110e04b55a, State: Initialized, Role: FOLLOWER
I20260810 14:40:31.183768 19215 consensus_queue.cc:260] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.184029 19215 ts_tablet_manager.cc:1428] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 14:40:31.308073 19221 raft_consensus.cc:482] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:31.308252 19221 raft_consensus.cc:504] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.309495 19221 leader_election.cc:288] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88a43a63d1f64217845294110e04b55a (127.15.84.1:34265), 271a88e76c214207baeddbe270db2b73 (127.15.84.2:38301)
I20260810 14:40:31.314074 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:31.310036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:48221 (local address 127.15.84.2:38301)
0810 14:40:31.310192 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:31.310197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:31.310435 (+ 238us) server_negotiation.cc:407] Connection header received
0810 14:40:31.310607 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:31.310610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:31.310651 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:31.310718 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:31.311014 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.311819 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.312976 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.313495 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.313602 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:31.313681 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:31.313861 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 14:40:31.313938 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:40:31.314127 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:31.309839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:38301 (local address 127.15.84.3:48221)
0810 14:40:31.310352 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:31.310364 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:31.310497 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:31.310745 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:31.310748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:31.310764 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:31.310903 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:31.310908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.311949 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.311953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:31.312841 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:31.312851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.313614 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.313618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:31.313733 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:31.313759 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:40:31.313836 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:40:31.313910 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":25,"thread_start_us":77,"threads_started":1}
I20260810 14:40:31.314796 19282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:31.309817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:34265 (local address 127.15.84.3:44509)
0810 14:40:31.310184 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:31.310199 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:31.310344 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:31.310724 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:31.310728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:31.310747 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:31.310894 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:31.310901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.312056 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.312059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:31.313294 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:31.313307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.314392 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.314396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:31.314495 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:31.314520 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:40:31.314590 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 14:40:31.314656 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":25,"spinlock_wait_cycles":16896,"thread_start_us":100,"threads_started":1}
I20260810 14:40:31.314747 18888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b387c453397d48f2a75232378e631af8" candidate_uuid: "e06b6bfa76674c9a9d828987d69162a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271a88e76c214207baeddbe270db2b73" is_pre_election: true
I20260810 14:40:31.314848 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:31.310255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:44509 (local address 127.15.84.1:34265)
0810 14:40:31.310391 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:31.310396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:31.310411 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:40:31.310471 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:31.310474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:31.310523 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:31.310599 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:31.311183 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.311920 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.313428 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.313992 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.314094 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:31.314160 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:31.314641 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 14:40:31.314730 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:40:31.314917 18888 raft_consensus.cc:2454] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e06b6bfa76674c9a9d828987d69162a8 in term 0.
I20260810 14:40:31.315330 19008 leader_election.cc:302] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 271a88e76c214207baeddbe270db2b73, e06b6bfa76674c9a9d828987d69162a8; no voters:
I20260810 14:40:31.315639 19221 raft_consensus.cc:2781] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:40:31.315601 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b387c453397d48f2a75232378e631af8" candidate_uuid: "e06b6bfa76674c9a9d828987d69162a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a43a63d1f64217845294110e04b55a" is_pre_election: true
I20260810 14:40:31.315708 19221 raft_consensus.cc:482] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:40:31.315743 19221 raft_consensus.cc:3037] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:31.315821 18670 raft_consensus.cc:2454] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e06b6bfa76674c9a9d828987d69162a8 in term 0.
I20260810 14:40:31.317057 19221 raft_consensus.cc:504] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.317505 19221 leader_election.cc:288] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [CANDIDATE]: Term 1 election: Requested vote from peers 88a43a63d1f64217845294110e04b55a (127.15.84.1:34265), 271a88e76c214207baeddbe270db2b73 (127.15.84.2:38301)
I20260810 14:40:31.317836 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b387c453397d48f2a75232378e631af8" candidate_uuid: "e06b6bfa76674c9a9d828987d69162a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a43a63d1f64217845294110e04b55a"
I20260810 14:40:31.317873 18888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b387c453397d48f2a75232378e631af8" candidate_uuid: "e06b6bfa76674c9a9d828987d69162a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271a88e76c214207baeddbe270db2b73"
I20260810 14:40:31.317950 18670 raft_consensus.cc:3037] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:31.317991 18888 raft_consensus.cc:3037] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:31.318941 18670 raft_consensus.cc:2454] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e06b6bfa76674c9a9d828987d69162a8 in term 1.
I20260810 14:40:31.319185 18888 raft_consensus.cc:2454] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e06b6bfa76674c9a9d828987d69162a8 in term 1.
I20260810 14:40:31.319252 19005 leader_election.cc:302] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a43a63d1f64217845294110e04b55a, e06b6bfa76674c9a9d828987d69162a8; no voters:
I20260810 14:40:31.319419 19221 raft_consensus.cc:2781] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:31.319623 19221 raft_consensus.cc:686] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [term 1 LEADER]: Becoming Leader. State: Replica: e06b6bfa76674c9a9d828987d69162a8, State: Running, Role: LEADER
I20260810 14:40:31.319912 19221 consensus_queue.cc:237] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:31.322525 18414 catalog_manager.cc:5495] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e06b6bfa76674c9a9d828987d69162a8 (127.15.84.3). New cstate: current_term: 1 leader_uuid: "e06b6bfa76674c9a9d828987d69162a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } health_report { overall_health: HEALTHY } } }
I20260810 14:40:31.346364 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:31.342795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50898 (local address 127.15.84.3:34533)
0810 14:40:31.342919 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:31.342923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:31.342938 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:40:31.343043 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:31.343046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:31.343101 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:31.343200 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:31.343437 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.343952 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.344812 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.345043 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.345131 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:31.345445 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:31.345951 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:31.346066 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:31.346140 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 14:40:31.346214 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 14:40:31.358474 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:31.354590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:59843 (local address 127.15.84.1:34265)
0810 14:40:31.354911 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:31.354917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:31.354932 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:40:31.354995 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:31.354998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:31.355050 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:31.355144 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:31.355415 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.356249 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.357258 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.357587 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.357698 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:31.357981 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:31.358078 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:31.358164 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:31.358246 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 14:40:31.358306 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260810 14:40:31.358978 19287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:31.354402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:34265 (local address 127.15.84.3:59843)
0810 14:40:31.354778 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:31.354792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:31.354876 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:31.355175 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:31.355178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:31.355187 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:40:31.355315 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:31.355321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.356365 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.356368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:31.357118 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:31.357129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.357697 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.357700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:31.357776 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:31.357829 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:40:31.358168 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:40:31.358183 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 14:40:31.358232 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 14:40:31.358309 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":97,"threads_started":1}
I20260810 14:40:31.365944 18888 raft_consensus.cc:1261] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Refusing update from remote peer e06b6bfa76674c9a9d828987d69162a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:31.366063 18670 raft_consensus.cc:1261] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Refusing update from remote peer e06b6bfa76674c9a9d828987d69162a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:31.366458 19221 consensus_queue.cc:1035] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:31.366673 19283 consensus_queue.cc:1035] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:31.389431 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:31.385540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:33403 (local address 127.15.84.3:34533)
0810 14:40:31.385822 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:31.385826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:31.385839 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:40:31.385880 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:31.385883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:31.385935 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:31.386004 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:31.386380 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.387133 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.388240 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.388601 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.388694 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:31.388933 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:31.389032 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:31.389138 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:31.389265 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 14:40:31.389321 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260810 14:40:31.389472 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:31.385435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:34533 (local address 127.15.84.1:33403)
0810 14:40:31.385558 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:31.385574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:31.385650 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:31.386127 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:31.386130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:31.386138 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:40:31.386257 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:31.386263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.387274 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.387278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:31.388110 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:31.388121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:31.388706 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:31.388709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:31.388778 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:31.388809 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:40:31.389138 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:40:31.389145 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:40:31.389241 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 14:40:31.389338 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 14:40:31.390237 19248 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 14:40:32.951500 18411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:32.951673 18411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:32.954465 18411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e: Sending DeleteTablet for 3 replicas of tablet b387c453397d48f2a75232378e631af8
I20260810 14:40:32.955179 18647 tablet_service.cc:1514] Processing DeleteTablet for tablet b387c453397d48f2a75232378e631af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:32 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:41938
I20260810 14:40:32.955258 19077 tablet_service.cc:1514] Processing DeleteTablet for tablet b387c453397d48f2a75232378e631af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:32 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:50884
I20260810 14:40:32.955770 18864 tablet_service.cc:1514] Processing DeleteTablet for tablet b387c453397d48f2a75232378e631af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:32 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:42322
I20260810 14:40:32.956360 19328 tablet_replica.cc:330] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a: stopping tablet replica
I20260810 14:40:32.956501 19328 raft_consensus.cc:2229] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:32.956630 19328 raft_consensus.cc:2258] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:32.956673 19330 tablet_replica.cc:330] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8: stopping tablet replica
I20260810 14:40:32.956795 19330 raft_consensus.cc:2229] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:40:32.957140 19330 raft_consensus.cc:2258] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:32.957484 19328 ts_tablet_manager.cc:1905] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:32.957888 19330 ts_tablet_manager.cc:1905] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:32.960455 19328 ts_tablet_manager.cc:1918] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:40:32.960455 19330 ts_tablet_manager.cc:1918] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:40:32.960551 19330 log.cc:1196] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/b387c453397d48f2a75232378e631af8
I20260810 14:40:32.960566 19328 log.cc:1196] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/wal/wals/b387c453397d48f2a75232378e631af8
I20260810 14:40:32.960775 19331 tablet_replica.cc:330] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73: stopping tablet replica
I20260810 14:40:32.960822 19328 ts_tablet_manager.cc:1939] T b387c453397d48f2a75232378e631af8 P 88a43a63d1f64217845294110e04b55a: Deleting consensus metadata
I20260810 14:40:32.960829 19330 ts_tablet_manager.cc:1939] T b387c453397d48f2a75232378e631af8 P e06b6bfa76674c9a9d828987d69162a8: Deleting consensus metadata
I20260810 14:40:32.960933 19331 raft_consensus.cc:2229] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:32.961237 19331 raft_consensus.cc:2258] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:32.961977 18389 catalog_manager.cc:4841] TS 88a43a63d1f64217845294110e04b55a (127.15.84.1:34265): tablet b387c453397d48f2a75232378e631af8 (table test-table [id=aaf8ee343da84f51b06928cd1d1556d0]) successfully deleted
I20260810 14:40:32.962023 19331 ts_tablet_manager.cc:1905] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:32.962932 18392 catalog_manager.cc:4841] TS e06b6bfa76674c9a9d828987d69162a8 (127.15.84.3:34533): tablet b387c453397d48f2a75232378e631af8 (table test-table [id=aaf8ee343da84f51b06928cd1d1556d0]) successfully deleted
I20260810 14:40:32.964284 19331 ts_tablet_manager.cc:1918] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:40:32.964358 19331 log.cc:1196] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/b387c453397d48f2a75232378e631af8
I20260810 14:40:32.964609 19331 ts_tablet_manager.cc:1939] T b387c453397d48f2a75232378e631af8 P 271a88e76c214207baeddbe270db2b73: Deleting consensus metadata
I20260810 14:40:32.965277 18390 catalog_manager.cc:4841] TS 271a88e76c214207baeddbe270db2b73 (127.15.84.2:38301): tablet b387c453397d48f2a75232378e631af8 (table test-table [id=aaf8ee343da84f51b06928cd1d1556d0]) successfully deleted
Aug 10 14:40:32 dist-test-slave-rmhg krb5kdc[18339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372832, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:40:32 dist-test-slave-rmhg krb5kdc[18339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372832, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:32.988960 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:32.979928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.15.84.62:41501)
0810 14:40:32.980228 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:32.980237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:32.980255 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:40:32.980403 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:32.980406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:32.980452 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:32.980530 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:32.981329 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:32.981914 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:32.982809 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:32.983050 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:32.983149 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:32.985689 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:32.985693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:32.985695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:32.985727 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:32.987264 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:32.987740 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:32.987744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:32.987746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:32.987845 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:32.988162 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:32.988164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:32.988166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:32.988365 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:32.988505 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:32.988673 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 14:40:32.988792 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":117,"threads_started":1}
I20260810 14:40:32.991640 18411 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:40:32.992072 18411 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:32.997172 18647 tablet_service.cc:1467] Processing CreateTablet for tablet 757c449ed2f2469a808d931593437620 (DEFAULT_TABLE table=test-table [id=439c3bd00c2e47a1be3540edc954c043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:32.997206 18864 tablet_service.cc:1467] Processing CreateTablet for tablet 757c449ed2f2469a808d931593437620 (DEFAULT_TABLE table=test-table [id=439c3bd00c2e47a1be3540edc954c043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:32.997365 19077 tablet_service.cc:1467] Processing CreateTablet for tablet 757c449ed2f2469a808d931593437620 (DEFAULT_TABLE table=test-table [id=439c3bd00c2e47a1be3540edc954c043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:32.997524 18864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757c449ed2f2469a808d931593437620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:32.997540 18647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757c449ed2f2469a808d931593437620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:32.997650 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757c449ed2f2469a808d931593437620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:33.000262 19344 tablet_bootstrap.cc:492] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73: Bootstrap starting.
I20260810 14:40:33.000285 19346 tablet_bootstrap.cc:492] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8: Bootstrap starting.
I20260810 14:40:33.000630 19343 tablet_bootstrap.cc:492] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a: Bootstrap starting.
I20260810 14:40:33.001245 19344 tablet_bootstrap.cc:654] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:33.001266 19346 tablet_bootstrap.cc:654] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:33.001470 19343 tablet_bootstrap.cc:654] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:33.002413 19344 tablet_bootstrap.cc:492] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73: No bootstrap required, opened a new log
I20260810 14:40:33.002426 19346 tablet_bootstrap.cc:492] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8: No bootstrap required, opened a new log
I20260810 14:40:33.002496 19346 ts_tablet_manager.cc:1397] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 14:40:33.002501 19344 ts_tablet_manager.cc:1397] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260810 14:40:33.002931 19346 raft_consensus.cc:348] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.002964 19344 raft_consensus.cc:348] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.003077 19346 raft_consensus.cc:374] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:33.003082 19344 raft_consensus.cc:374] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:33.003124 19344 raft_consensus.cc:729] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271a88e76c214207baeddbe270db2b73, State: Initialized, Role: FOLLOWER
I20260810 14:40:33.003121 19346 raft_consensus.cc:729] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06b6bfa76674c9a9d828987d69162a8, State: Initialized, Role: FOLLOWER
I20260810 14:40:33.003253 19344 consensus_queue.cc:260] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.003255 19346 consensus_queue.cc:260] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.003536 19346 ts_tablet_manager.cc:1428] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:40:33.003639 19344 ts_tablet_manager.cc:1428] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:40:33.004068 19343 tablet_bootstrap.cc:492] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a: No bootstrap required, opened a new log
I20260810 14:40:33.004174 19343 ts_tablet_manager.cc:1397] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 14:40:33.004683 19343 raft_consensus.cc:348] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.004812 19343 raft_consensus.cc:374] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:33.004844 19343 raft_consensus.cc:729] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a43a63d1f64217845294110e04b55a, State: Initialized, Role: FOLLOWER
I20260810 14:40:33.005008 19343 consensus_queue.cc:260] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.005810 19343 ts_tablet_manager.cc:1428] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:40:33.289819 19351 raft_consensus.cc:482] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:33.289990 19351 raft_consensus.cc:504] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.291134 19351 leader_election.cc:288] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88a43a63d1f64217845294110e04b55a (127.15.84.1:34265), e06b6bfa76674c9a9d828987d69162a8 (127.15.84.3:34533)
I20260810 14:40:33.295280 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:33.291525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:40541 (local address 127.15.84.3:34533)
0810 14:40:33.291858 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:33.291867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:33.291884 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:40:33.292086 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:33.292089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:33.292133 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:33.292211 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:33.292524 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.293067 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.294270 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.294694 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.294782 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:33.294858 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:33.295047 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 14:40:33.295132 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
I20260810 14:40:33.295297 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:33.291343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:34265 (local address 127.15.84.2:46115)
0810 14:40:33.291774 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:33.291791 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:33.291962 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:33.292240 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:33.292243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:33.292259 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:33.292395 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:33.292401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.293242 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.293245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:33.294150 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:33.294161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.294850 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.294853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:33.294952 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:33.294971 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 14:40:33.295044 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 14:40:33.295129 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":40,"spinlock_wait_cycles":35968,"thread_start_us":136,"threads_started":1}
I20260810 14:40:33.295292 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:33.291462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:46115 (local address 127.15.84.1:34265)
0810 14:40:33.291779 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:33.291789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:33.291887 (+ 98us) server_negotiation.cc:407] Connection header received
0810 14:40:33.292073 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:33.292075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:33.292121 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:33.292199 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:33.292507 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.293082 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.294282 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.294723 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.294830 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:33.294902 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:33.295083 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 14:40:33.295165 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":97,"threads_started":1}
I20260810 14:40:33.295306 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:33.291407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:34533 (local address 127.15.84.2:40541)
0810 14:40:33.291813 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:33.291827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:33.291962 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:33.292228 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:33.292231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:33.292247 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:33.292408 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:33.292414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.293205 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.293209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:33.294135 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:33.294145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.294811 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.294815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:33.294914 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:33.294939 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:40:33.295007 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 14:40:33.295089 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":43,"spinlock_wait_cycles":7040,"thread_start_us":118,"threads_started":1}
I20260810 14:40:33.295811 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757c449ed2f2469a808d931593437620" candidate_uuid: "271a88e76c214207baeddbe270db2b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a43a63d1f64217845294110e04b55a" is_pre_election: true
I20260810 14:40:33.295984 18670 raft_consensus.cc:2454] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 271a88e76c214207baeddbe270db2b73 in term 0.
I20260810 14:40:33.296157 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757c449ed2f2469a808d931593437620" candidate_uuid: "271a88e76c214207baeddbe270db2b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e06b6bfa76674c9a9d828987d69162a8" is_pre_election: true
I20260810 14:40:33.296324 19106 raft_consensus.cc:2454] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 271a88e76c214207baeddbe270db2b73 in term 0.
I20260810 14:40:33.296437 18798 leader_election.cc:302] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 271a88e76c214207baeddbe270db2b73, 88a43a63d1f64217845294110e04b55a; no voters:
I20260810 14:40:33.296737 19351 raft_consensus.cc:2781] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:40:33.296795 19351 raft_consensus.cc:482] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:40:33.296821 19351 raft_consensus.cc:3037] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:33.297917 19351 raft_consensus.cc:504] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.298275 19351 leader_election.cc:288] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [CANDIDATE]: Term 1 election: Requested vote from peers 88a43a63d1f64217845294110e04b55a (127.15.84.1:34265), e06b6bfa76674c9a9d828987d69162a8 (127.15.84.3:34533)
I20260810 14:40:33.298614 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757c449ed2f2469a808d931593437620" candidate_uuid: "271a88e76c214207baeddbe270db2b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e06b6bfa76674c9a9d828987d69162a8"
I20260810 14:40:33.298656 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757c449ed2f2469a808d931593437620" candidate_uuid: "271a88e76c214207baeddbe270db2b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a43a63d1f64217845294110e04b55a"
I20260810 14:40:33.298731 19106 raft_consensus.cc:3037] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:33.298777 18670 raft_consensus.cc:3037] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:33.299651 19106 raft_consensus.cc:2454] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 271a88e76c214207baeddbe270db2b73 in term 1.
I20260810 14:40:33.299790 18670 raft_consensus.cc:2454] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 271a88e76c214207baeddbe270db2b73 in term 1.
I20260810 14:40:33.299968 18801 leader_election.cc:302] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 271a88e76c214207baeddbe270db2b73, e06b6bfa76674c9a9d828987d69162a8; no voters:
I20260810 14:40:33.300144 19351 raft_consensus.cc:2781] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:33.300392 19351 raft_consensus.cc:686] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [term 1 LEADER]: Becoming Leader. State: Replica: 271a88e76c214207baeddbe270db2b73, State: Running, Role: LEADER
I20260810 14:40:33.300689 19351 consensus_queue.cc:237] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.303208 18411 catalog_manager.cc:5495] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 271a88e76c214207baeddbe270db2b73 (127.15.84.2). New cstate: current_term: 1 leader_uuid: "271a88e76c214207baeddbe270db2b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } health_report { overall_health: UNKNOWN } } }
I20260810 14:40:33.374236 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:33.370409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.15.84.2:38301)
0810 14:40:33.370686 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:33.370695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:33.370708 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:40:33.370791 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:33.370793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:33.370847 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:33.370957 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:33.371211 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.371782 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.372625 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.372879 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.372972 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:33.373301 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:33.373822 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:33.373929 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:33.374003 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 14:40:33.374064 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
I20260810 14:40:33.385521 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:33.381786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:34265 (local address 127.15.84.2:55383)
0810 14:40:33.382286 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:33.382299 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:33.382409 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:33.382668 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:33.382671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:33.382679 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:40:33.382812 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:33.382818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.383581 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.383585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:33.384245 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:33.384255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.384714 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.384718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:33.384822 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:33.384878 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:40:33.385200 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:40:33.385207 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:40:33.385268 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 14:40:33.385358 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":128,"threads_started":1}
I20260810 14:40:33.385514 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:33.382039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:55383 (local address 127.15.84.1:34265)
0810 14:40:33.382189 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:33.382194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:33.382368 (+ 174us) server_negotiation.cc:407] Connection header received
0810 14:40:33.382531 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:33.382534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:33.382586 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:33.382647 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:33.382913 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.383432 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.384365 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.384608 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.384700 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:33.385012 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:33.385094 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:33.385182 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:33.385288 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 14:40:33.385350 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:40:33.392323 19106 raft_consensus.cc:1261] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Refusing update from remote peer 271a88e76c214207baeddbe270db2b73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:33.392432 18670 raft_consensus.cc:1261] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Refusing update from remote peer 271a88e76c214207baeddbe270db2b73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:33.392903 19359 consensus_queue.cc:1035] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:33.393105 19351 consensus_queue.cc:1035] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:33.397655 19300 mvcc.cc:205] Tried to move back new op lower bound from 7316983125570863104 to 7316983125202632704. Current Snapshot: MvccSnapshot[applied={T|T < 7316983125570863104}]
I20260810 14:40:33.410955 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:33.407516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:50845 (local address 127.15.84.2:38301)
0810 14:40:33.407638 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:33.407643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:33.407655 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:40:33.407695 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:33.407698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:33.407738 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:33.407797 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:33.408046 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.408606 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.409896 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.410176 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.410279 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:33.410563 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:33.410650 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:33.410708 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:33.410793 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 14:40:33.410846 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:40:33.411013 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:33.407163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:38301 (local address 127.15.84.1:50845)
0810 14:40:33.407466 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:33.407479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:33.407560 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:33.407816 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:33.407818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:33.407826 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:40:33.407935 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:33.407940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.408771 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.408776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:33.409769 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:33.409779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.410279 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.410283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:33.410365 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:33.410398 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:40:33.410716 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:40:33.410722 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:40:33.410786 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 14:40:33.410874 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":124,"threads_started":1}
I20260810 14:40:33.435333 18418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:33.435497 18418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:33.437793 18418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e: Sending DeleteTablet for 3 replicas of tablet 757c449ed2f2469a808d931593437620
I20260810 14:40:33.438473 18647 tablet_service.cc:1514] Processing DeleteTablet for tablet 757c449ed2f2469a808d931593437620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:33 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:41938
I20260810 14:40:33.438485 18864 tablet_service.cc:1514] Processing DeleteTablet for tablet 757c449ed2f2469a808d931593437620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:33 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:42322
I20260810 14:40:33.438692 19328 tablet_replica.cc:330] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a: stopping tablet replica
I20260810 14:40:33.438710 19331 tablet_replica.cc:330] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73: stopping tablet replica
I20260810 14:40:33.438799 19328 raft_consensus.cc:2229] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:33.438838 19331 raft_consensus.cc:2229] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:40:33.438898 19328 raft_consensus.cc:2258] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:33.439167 19331 raft_consensus.cc:2258] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:33.439391 19077 tablet_service.cc:1514] Processing DeleteTablet for tablet 757c449ed2f2469a808d931593437620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:33 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:50884
I20260810 14:40:33.439796 19330 tablet_replica.cc:330] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8: stopping tablet replica
I20260810 14:40:33.439857 19328 ts_tablet_manager.cc:1905] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:33.439934 19330 raft_consensus.cc:2229] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:33.440068 19330 raft_consensus.cc:2258] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:33.440897 19331 ts_tablet_manager.cc:1905] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:33.441632 19330 ts_tablet_manager.cc:1905] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:33.442553 19328 ts_tablet_manager.cc:1918] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:40:33.442624 19328 log.cc:1196] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-0/wal/wals/757c449ed2f2469a808d931593437620
I20260810 14:40:33.442878 19328 ts_tablet_manager.cc:1939] T 757c449ed2f2469a808d931593437620 P 88a43a63d1f64217845294110e04b55a: Deleting consensus metadata
I20260810 14:40:33.443612 19331 ts_tablet_manager.cc:1918] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:40:33.443682 19331 log.cc:1196] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/757c449ed2f2469a808d931593437620
I20260810 14:40:33.443795 18389 catalog_manager.cc:4841] TS 88a43a63d1f64217845294110e04b55a (127.15.84.1:34265): tablet 757c449ed2f2469a808d931593437620 (table test-table [id=439c3bd00c2e47a1be3540edc954c043]) successfully deleted
I20260810 14:40:33.443931 19331 ts_tablet_manager.cc:1939] T 757c449ed2f2469a808d931593437620 P 271a88e76c214207baeddbe270db2b73: Deleting consensus metadata
I20260810 14:40:33.444160 19330 ts_tablet_manager.cc:1918] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:40:33.444223 19330 log.cc:1196] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/757c449ed2f2469a808d931593437620
I20260810 14:40:33.444382 19330 ts_tablet_manager.cc:1939] T 757c449ed2f2469a808d931593437620 P e06b6bfa76674c9a9d828987d69162a8: Deleting consensus metadata
I20260810 14:40:33.444696 18390 catalog_manager.cc:4841] TS 271a88e76c214207baeddbe270db2b73 (127.15.84.2:38301): tablet 757c449ed2f2469a808d931593437620 (table test-table [id=439c3bd00c2e47a1be3540edc954c043]) successfully deleted
I20260810 14:40:33.445358 18392 catalog_manager.cc:4841] TS e06b6bfa76674c9a9d828987d69162a8 (127.15.84.3:34533): tablet 757c449ed2f2469a808d931593437620 (table test-table [id=439c3bd00c2e47a1be3540edc954c043]) successfully deleted
I20260810 14:40:33.445873 18418 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:40:33.446349 18418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:33.451210 18864 tablet_service.cc:1467] Processing CreateTablet for tablet b5014313bbe94ed48348ca1fce709b77 (DEFAULT_TABLE table=test-table [id=51558aeeb611440e8b85aa508534b0f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:33.451371 19077 tablet_service.cc:1467] Processing CreateTablet for tablet b5014313bbe94ed48348ca1fce709b77 (DEFAULT_TABLE table=test-table [id=51558aeeb611440e8b85aa508534b0f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:33.451563 18864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5014313bbe94ed48348ca1fce709b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:33.451704 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5014313bbe94ed48348ca1fce709b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:33.453647 19344 tablet_bootstrap.cc:492] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73: Bootstrap starting.
I20260810 14:40:33.454198 18647 tablet_service.cc:1467] Processing CreateTablet for tablet b5014313bbe94ed48348ca1fce709b77 (DEFAULT_TABLE table=test-table [id=51558aeeb611440e8b85aa508534b0f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:33.454485 18647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5014313bbe94ed48348ca1fce709b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:33.454514 19346 tablet_bootstrap.cc:492] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8: Bootstrap starting.
I20260810 14:40:33.454712 19344 tablet_bootstrap.cc:654] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:33.455397 19346 tablet_bootstrap.cc:654] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:33.456660 19346 tablet_bootstrap.cc:492] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8: No bootstrap required, opened a new log
I20260810 14:40:33.456750 19346 ts_tablet_manager.cc:1397] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 14:40:33.456779 19343 tablet_bootstrap.cc:492] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a: Bootstrap starting.
I20260810 14:40:33.457190 19346 raft_consensus.cc:348] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.457309 19346 raft_consensus.cc:374] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:33.457340 19346 raft_consensus.cc:729] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06b6bfa76674c9a9d828987d69162a8, State: Initialized, Role: FOLLOWER
I20260810 14:40:33.457473 19346 consensus_queue.cc:260] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.457659 19343 tablet_bootstrap.cc:654] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:33.457779 19346 ts_tablet_manager.cc:1428] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:33.458678 19344 tablet_bootstrap.cc:492] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73: No bootstrap required, opened a new log
I20260810 14:40:33.458789 19344 ts_tablet_manager.cc:1397] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260810 14:40:33.459234 19344 raft_consensus.cc:348] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.459339 19344 raft_consensus.cc:374] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:33.459368 19344 raft_consensus.cc:729] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271a88e76c214207baeddbe270db2b73, State: Initialized, Role: FOLLOWER
I20260810 14:40:33.459512 19344 consensus_queue.cc:260] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.459805 19344 ts_tablet_manager.cc:1428] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 14:40:33.460822 19343 tablet_bootstrap.cc:492] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a: No bootstrap required, opened a new log
I20260810 14:40:33.460925 19343 ts_tablet_manager.cc:1397] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 14:40:33.461421 19343 raft_consensus.cc:348] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.461551 19343 raft_consensus.cc:374] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:33.461585 19343 raft_consensus.cc:729] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a43a63d1f64217845294110e04b55a, State: Initialized, Role: FOLLOWER
I20260810 14:40:33.461719 19343 consensus_queue.cc:260] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.461983 19343 ts_tablet_manager.cc:1428] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:33.485018 19318 raft_consensus.cc:482] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:33.485196 19318 raft_consensus.cc:504] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.485631 19318 leader_election.cc:288] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88a43a63d1f64217845294110e04b55a (127.15.84.1:34265), 271a88e76c214207baeddbe270db2b73 (127.15.84.2:38301)
I20260810 14:40:33.486063 18888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5014313bbe94ed48348ca1fce709b77" candidate_uuid: "e06b6bfa76674c9a9d828987d69162a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271a88e76c214207baeddbe270db2b73" is_pre_election: true
I20260810 14:40:33.486066 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5014313bbe94ed48348ca1fce709b77" candidate_uuid: "e06b6bfa76674c9a9d828987d69162a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a43a63d1f64217845294110e04b55a" is_pre_election: true
I20260810 14:40:33.486196 18888 raft_consensus.cc:2454] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e06b6bfa76674c9a9d828987d69162a8 in term 0.
I20260810 14:40:33.486239 18670 raft_consensus.cc:2454] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e06b6bfa76674c9a9d828987d69162a8 in term 0.
I20260810 14:40:33.486526 19008 leader_election.cc:302] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 271a88e76c214207baeddbe270db2b73, e06b6bfa76674c9a9d828987d69162a8; no voters:
I20260810 14:40:33.486756 19318 raft_consensus.cc:2781] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:40:33.486828 19318 raft_consensus.cc:482] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:40:33.486867 19318 raft_consensus.cc:3037] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:33.488209 19318 raft_consensus.cc:504] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.488612 19318 leader_election.cc:288] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [CANDIDATE]: Term 1 election: Requested vote from peers 88a43a63d1f64217845294110e04b55a (127.15.84.1:34265), 271a88e76c214207baeddbe270db2b73 (127.15.84.2:38301)
I20260810 14:40:33.488984 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5014313bbe94ed48348ca1fce709b77" candidate_uuid: "e06b6bfa76674c9a9d828987d69162a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a43a63d1f64217845294110e04b55a"
I20260810 14:40:33.489115 18670 raft_consensus.cc:3037] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:33.490171 18670 raft_consensus.cc:2454] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e06b6bfa76674c9a9d828987d69162a8 in term 1.
I20260810 14:40:33.490512 19005 leader_election.cc:302] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a43a63d1f64217845294110e04b55a, e06b6bfa76674c9a9d828987d69162a8; no voters:
I20260810 14:40:33.490736 19318 raft_consensus.cc:2781] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:33.490916 19318 raft_consensus.cc:686] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [term 1 LEADER]: Becoming Leader. State: Replica: e06b6bfa76674c9a9d828987d69162a8, State: Running, Role: LEADER
I20260810 14:40:33.491063 19318 consensus_queue.cc:237] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } }
I20260810 14:40:33.490592 18888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5014313bbe94ed48348ca1fce709b77" candidate_uuid: "e06b6bfa76674c9a9d828987d69162a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271a88e76c214207baeddbe270db2b73"
I20260810 14:40:33.491942 18888 raft_consensus.cc:3037] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:33.492712 18418 catalog_manager.cc:5495] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e06b6bfa76674c9a9d828987d69162a8 (127.15.84.3). New cstate: current_term: 1 leader_uuid: "e06b6bfa76674c9a9d828987d69162a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e06b6bfa76674c9a9d828987d69162a8" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34533 } health_report { overall_health: HEALTHY } } }
I20260810 14:40:33.493196 18888 raft_consensus.cc:2454] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e06b6bfa76674c9a9d828987d69162a8 in term 1.
I20260810 14:40:33.515224 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:33.511951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.15.84.3:34533)
0810 14:40:33.512074 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:33.512078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:33.512102 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:40:33.512173 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:33.512176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:33.512231 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:33.512313 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:33.512501 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.513337 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.514152 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:33.514499 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:33.514602 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:33.514787 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:33.514876 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:33.514939 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:33.515005 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 14:40:33.515065 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:40:33.522222 18890 raft_consensus.cc:1261] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Refusing update from remote peer e06b6bfa76674c9a9d828987d69162a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:33.522586 18670 raft_consensus.cc:1261] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Refusing update from remote peer e06b6bfa76674c9a9d828987d69162a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:33.522588 19318 consensus_queue.cc:1035] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "271a88e76c214207baeddbe270db2b73" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:33.523167 19318 consensus_queue.cc:1035] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a43a63d1f64217845294110e04b55a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 34265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:40:33.538573 19300 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 14:40:35.096979 18411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:35.097198 18411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:35.099893 18411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d275de245cf44508b6f355d826287a4e: Sending DeleteTablet for 3 replicas of tablet b5014313bbe94ed48348ca1fce709b77
I20260810 14:40:35.100606 18647 tablet_service.cc:1514] Processing DeleteTablet for tablet b5014313bbe94ed48348ca1fce709b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:35 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:41938
I20260810 14:40:35.100644 18864 tablet_service.cc:1514] Processing DeleteTablet for tablet b5014313bbe94ed48348ca1fce709b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:35 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:42322
I20260810 14:40:35.100644 19077 tablet_service.cc:1514] Processing DeleteTablet for tablet b5014313bbe94ed48348ca1fce709b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:35 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:50884
I20260810 14:40:35.101007 19425 tablet_replica.cc:330] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a: stopping tablet replica
I20260810 14:40:35.101042 19428 tablet_replica.cc:330] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8: stopping tablet replica
I20260810 14:40:35.101190 19425 raft_consensus.cc:2229] T b5014313bbe94ed48348ca1fce709b77 P 88a43a63d1f64217845294110e04b55a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:35.101205 19428 raft_consensus.cc:2229] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:40:35.101356 19427 tablet_replica.cc:330] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73: stopping tablet replica
I20260810 14:40:35.101428 19428 raft_consensus.cc:2258] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:35.101442 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 18536
I20260810 14:40:35.101498 19427 raft_consensus.cc:2229] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:35.101611 19427 raft_consensus.cc:2258] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:35.121982 19428 ts_tablet_manager.cc:1905] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:35.128156 19427 ts_tablet_manager.cc:1905] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:35.129784 19428 ts_tablet_manager.cc:1918] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:40:35.129905 19428 log.cc:1196] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/b5014313bbe94ed48348ca1fce709b77
I20260810 14:40:35.130246 19428 ts_tablet_manager.cc:1939] T b5014313bbe94ed48348ca1fce709b77 P e06b6bfa76674c9a9d828987d69162a8: Deleting consensus metadata
I20260810 14:40:35.131047 19427 ts_tablet_manager.cc:1918] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:40:35.131134 19427 log.cc:1196] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/b5014313bbe94ed48348ca1fce709b77
I20260810 14:40:35.131280 18392 catalog_manager.cc:4841] TS e06b6bfa76674c9a9d828987d69162a8 (127.15.84.3:34533): tablet b5014313bbe94ed48348ca1fce709b77 (table test-table [id=51558aeeb611440e8b85aa508534b0f8]) successfully deleted
I20260810 14:40:35.131368 19427 ts_tablet_manager.cc:1939] T b5014313bbe94ed48348ca1fce709b77 P 271a88e76c214207baeddbe270db2b73: Deleting consensus metadata
I20260810 14:40:35.132205 18390 catalog_manager.cc:4841] TS 271a88e76c214207baeddbe270db2b73 (127.15.84.2:38301): tablet b5014313bbe94ed48348ca1fce709b77 (table test-table [id=51558aeeb611440e8b85aa508534b0f8]) successfully deleted
W20260810 14:40:35.136459 18389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:34265 (error 108)
I20260810 14:40:35.137454 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 18763
I20260810 14:40:35.141191 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:35.137298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:34265 (local address 127.0.0.1:41956)
0810 14:40:35.140886 (+ 3588us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:35.141024 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:34265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
W20260810 14:40:35.141471 18389 catalog_manager.cc:4584] TS 88a43a63d1f64217845294110e04b55a (127.15.84.1:34265): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5014313bbe94ed48348ca1fce709b77: Network error: Client connection negotiation failed: client connection to 127.15.84.1:34265: connect: Connection refused (error 111)
I20260810 14:40:35.148531 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 18981
I20260810 14:40:35.165423 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 18358
2026-08-10T14:40:35Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7404 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:35 dist-test-slave-rmhg krb5kdc[19437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:35 dist-test-slave-rmhg krb5kdc[19437](info): set up 2 sockets
Aug 10 14:40:35 dist-test-slave-rmhg krb5kdc[19437](info): commencing operation
krb5kdc: starting...
W20260810 14:40:36.324250 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:40:36 dist-test-slave-rmhg krb5kdc[19437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:40:36.455637 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:45123
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:45123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:36.575623 19457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:36.575942 19457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:36.576002 19457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:36.580242 19457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:36.580340 19457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:36.580369 19457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:36.580391 19457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:36.580410 19457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:36.586265 19457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:45123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:45123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:36.587527 19457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:36.588682 19457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:36.597667 19471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:36.597779 19469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:36.597702 19466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:36.598337 19457 server_base.cc:1034] running on GCE node
I20260810 14:40:36.598980 19457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:36.600227 19457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:36.601399 19457 hybrid_clock.cc:648] HybridClock initialized: now 1786372836601384 us; error 34 us; skew 500 ppm
Aug 10 14:40:36 dist-test-slave-rmhg krb5kdc[19437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372836, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:36.604348 19457 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:40:36.605409 19457 webserver.cc:460] Webserver started at http://127.15.84.62:40689/ using document root <none> and password file <none>
I20260810 14:40:36.605988 19457 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:36.606051 19457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:36.606249 19457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:36.608453 19457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "43664edc216f42f1a4352456fb0934a0"
format_stamp: "Formatted at 2026-08-10 14:40:36 on dist-test-slave-rmhg"
server_key: "a44590a3d88e7621fc5ddc537de06143"
server_key_iv: "cf867d270ebd4b51af6ba2c979776143"
server_key_version: "encryptionkey@0"
I20260810 14:40:36.609020 19457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "43664edc216f42f1a4352456fb0934a0"
format_stamp: "Formatted at 2026-08-10 14:40:36 on dist-test-slave-rmhg"
server_key: "a44590a3d88e7621fc5ddc537de06143"
server_key_iv: "cf867d270ebd4b51af6ba2c979776143"
server_key_version: "encryptionkey@0"
I20260810 14:40:36.613085 19457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 14:40:36.615780 19480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:36.617131 19457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 14:40:36.617339 19457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "43664edc216f42f1a4352456fb0934a0"
format_stamp: "Formatted at 2026-08-10 14:40:36 on dist-test-slave-rmhg"
server_key: "a44590a3d88e7621fc5ddc537de06143"
server_key_iv: "cf867d270ebd4b51af6ba2c979776143"
server_key_version: "encryptionkey@0"
I20260810 14:40:36.617456 19457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:36.629326 19457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:36.632769 19457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:36.632975 19457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:36.640599 19457 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:45123
I20260810 14:40:36.640622 19558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:45123 every 8 connection(s)
I20260810 14:40:36.641700 19457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:36.643714 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 19457
I20260810 14:40:36.643863 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:36.644200 15696 external_mini_cluster.cc:1428] Setting key 8e6fba89f2a45c0bd677f67957ca4b69
I20260810 14:40:36.645314 19559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:36.652612 19559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0: Bootstrap starting.
Aug 10 14:40:36 dist-test-slave-rmhg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:36.655422 19559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:36.656226 19559 log.cc:824] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:36.658515 19559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0: No bootstrap required, opened a new log
I20260810 14:40:36.659420 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:36.645781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.15.84.62:45123)
0810 14:40:36.646222 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:36.646239 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:36.646278 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:40:36.646953 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:36.646971 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:36.647344 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:36.647648 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:36.648619 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:36.649748 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:36.650936 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:36.651454 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:36.651574 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:36.654542 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:36.654548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:36.654563 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:36.654599 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:36.656288 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:36.656779 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:36.656785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:36.656790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:36.656908 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:36.657254 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:36.657258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:36.657260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:36.657702 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:36.657934 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:36.658226 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 14:40:36.658577 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
I20260810 14:40:36.662725 19559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43664edc216f42f1a4352456fb0934a0" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45123 } }
I20260810 14:40:36.663018 19559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:36.663086 19559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43664edc216f42f1a4352456fb0934a0, State: Initialized, Role: FOLLOWER
I20260810 14:40:36.663539 19559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43664edc216f42f1a4352456fb0934a0" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45123 } }
I20260810 14:40:36.663691 19559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:36.663761 19559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:36.663872 19559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:36.664942 19559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43664edc216f42f1a4352456fb0934a0" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45123 } }
I20260810 14:40:36.665362 19559 leader_election.cc:302] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43664edc216f42f1a4352456fb0934a0; no voters:
I20260810 14:40:36.665791 19559 leader_election.cc:288] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:36.665894 19567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:36.666136 19567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [term 1 LEADER]: Becoming Leader. State: Replica: 43664edc216f42f1a4352456fb0934a0, State: Running, Role: LEADER
I20260810 14:40:36.666504 19567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43664edc216f42f1a4352456fb0934a0" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45123 } }
I20260810 14:40:36.667199 19559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:36.668298 19572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43664edc216f42f1a4352456fb0934a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43664edc216f42f1a4352456fb0934a0" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45123 } } }
I20260810 14:40:36.668330 19571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43664edc216f42f1a4352456fb0934a0. Latest consensus state: current_term: 1 leader_uuid: "43664edc216f42f1a4352456fb0934a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43664edc216f42f1a4352456fb0934a0" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45123 } } }
I20260810 14:40:36.668465 19571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:36.668465 19572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:36.668913 19581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:36.672163 19581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:36.678193 19581 catalog_manager.cc:1347] Generated new cluster ID: 1e2e99c2e2bf4f358fb95d9845422f63
I20260810 14:40:36.678289 19581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:36.689298 19581 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:36.690315 19581 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:36.695295 19581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43664edc216f42f1a4352456fb0934a0: Generated new TSK 0
I20260810 14:40:36.696237 19581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:40:36.735747 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:45123
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:36.856961 19596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:36.857347 19596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:36.857424 19596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:36.861626 19596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:36.861709 19596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:36.861794 19596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:40:36.867635 19596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:45123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:36.868927 19596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:36.870046 19596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:36.879961 19604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:36.879966 19606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:36.879983 19608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:36.880362 19596 server_base.cc:1034] running on GCE node
I20260810 14:40:36.880777 19596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:36.881492 19596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:36.882690 19596 hybrid_clock.cc:648] HybridClock initialized: now 1786372836882672 us; error 35 us; skew 500 ppm
Aug 10 14:40:36 dist-test-slave-rmhg krb5kdc[19437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372836, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:36.885610 19596 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:40:36.886567 19596 webserver.cc:460] Webserver started at http://127.15.84.1:37765/ using document root <none> and password file <none>
I20260810 14:40:36.887156 19596 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:36.887229 19596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:36.887431 19596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:36.889647 19596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "518210b8c6614cf8bf9e94642e9ca1ca"
format_stamp: "Formatted at 2026-08-10 14:40:36 on dist-test-slave-rmhg"
server_key: "45abb6edbca5e63b74797543decba5c4"
server_key_iv: "79f905a5895661e2dcff70e48d6d2c15"
server_key_version: "encryptionkey@0"
I20260810 14:40:36.890208 19596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "518210b8c6614cf8bf9e94642e9ca1ca"
format_stamp: "Formatted at 2026-08-10 14:40:36 on dist-test-slave-rmhg"
server_key: "45abb6edbca5e63b74797543decba5c4"
server_key_iv: "79f905a5895661e2dcff70e48d6d2c15"
server_key_version: "encryptionkey@0"
I20260810 14:40:36.894104 19596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:40:36.896801 19619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:36.898146 19596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:36.898281 19596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "518210b8c6614cf8bf9e94642e9ca1ca"
format_stamp: "Formatted at 2026-08-10 14:40:36 on dist-test-slave-rmhg"
server_key: "45abb6edbca5e63b74797543decba5c4"
server_key_iv: "79f905a5895661e2dcff70e48d6d2c15"
server_key_version: "encryptionkey@0"
I20260810 14:40:36.898394 19596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:36.913559 19596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:36.917490 19596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:36.917692 19596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:36.918335 19596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:36.919219 19596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:36.919269 19596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:36.919324 19596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:36.919351 19596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:36.928882 19596 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:40467
I20260810 14:40:36.928913 19789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:40467 every 8 connection(s)
I20260810 14:40:36.929910 19596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:40:36.934288 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 19596
I20260810 14:40:36.934428 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:40:36.934706 15696 external_mini_cluster.cc:1428] Setting key 6f819cc7968fcc115e535f69f4e18fee
Aug 10 14:40:36 dist-test-slave-rmhg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372836, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:36.945420 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:36.931865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:60235 (local address 127.15.84.62:45123)
0810 14:40:36.932016 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:36.932021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:36.932947 (+ 926us) server_negotiation.cc:407] Connection header received
0810 14:40:36.934237 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:36.934241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:36.934302 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:36.934403 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:36.935892 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:36.936612 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:36.937890 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:36.938212 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:36.938320 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:36.941238 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:36.941243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:36.941246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:36.941274 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:36.942848 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:36.943524 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:36.943527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:36.943528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:36.943605 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:36.944021 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:36.944023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:36.944025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:36.944219 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:36.944292 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:36.945074 (+ 782us) server_negotiation.cc:299] Negotiation successful
0810 14:40:36.945280 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:40:36.946533 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:36.932298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:45123 (local address 127.15.84.1:60235)
0810 14:40:36.932783 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:36.932830 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:36.934018 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:36.934622 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:36.934629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:36.935038 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:36.935733 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:36.935755 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:36.936772 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:36.936775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:36.937768 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:36.937779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:36.938385 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:36.938390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:36.938595 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:36.939226 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:36.939292 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:36.941020 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:36.942989 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:36.942997 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:36.943018 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:36.943423 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:36.943707 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:36.943711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:36.943712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:36.943929 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:36.944321 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:36.944328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:36.944744 (+ 416us) client_negotiation.cc:769] Sending connection context
0810 14:40:36.944984 (+ 240us) client_negotiation.cc:240] Negotiation successful
0810 14:40:36.945442 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20260810 14:40:36.947836 19790 heartbeater.cc:347] Connected to a master server at 127.15.84.62:45123
I20260810 14:40:36.948088 19790 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:36.948604 19790 heartbeater.cc:511] Master 127.15.84.62:45123 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
I20260810 14:40:36.950227 19506 ts_manager.cc:194] Registered new tserver with Master: 518210b8c6614cf8bf9e94642e9ca1ca (127.15.84.1:40467)
I20260810 14:40:36.951766 19506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:60235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:40:36.964581 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:45123
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:37.084685 19797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:37.085012 19797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:37.085067 19797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:37.089339 19797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:37.089426 19797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:37.089504 19797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:40:37.095345 19797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:45123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:37.096576 19797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:37.097752 19797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:37.108338 19803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:37.108464 19797 server_base.cc:1034] running on GCE node
W20260810 14:40:37.108376 19802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:37.108377 19805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:37.109144 19797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:37.109954 19797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:37.111115 19797 hybrid_clock.cc:648] HybridClock initialized: now 1786372837111090 us; error 40 us; skew 500 ppm
Aug 10 14:40:37 dist-test-slave-rmhg krb5kdc[19437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372837, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:37.114162 19797 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:40:37.115150 19797 webserver.cc:460] Webserver started at http://127.15.84.2:38607/ using document root <none> and password file <none>
I20260810 14:40:37.115768 19797 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:37.115842 19797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:37.116070 19797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:37.118288 19797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "054ee4a6e1ee417fad851ecb56873ffb"
format_stamp: "Formatted at 2026-08-10 14:40:37 on dist-test-slave-rmhg"
server_key: "d05d9a2ad6566ab01c007559e9080124"
server_key_iv: "b4f5be195faf07059f2cd0ca0026c1c2"
server_key_version: "encryptionkey@0"
I20260810 14:40:37.118854 19797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "054ee4a6e1ee417fad851ecb56873ffb"
format_stamp: "Formatted at 2026-08-10 14:40:37 on dist-test-slave-rmhg"
server_key: "d05d9a2ad6566ab01c007559e9080124"
server_key_iv: "b4f5be195faf07059f2cd0ca0026c1c2"
server_key_version: "encryptionkey@0"
I20260810 14:40:37.122947 19797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:40:37.125559 19814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:37.126828 19797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:40:37.126951 19797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "054ee4a6e1ee417fad851ecb56873ffb"
format_stamp: "Formatted at 2026-08-10 14:40:37 on dist-test-slave-rmhg"
server_key: "d05d9a2ad6566ab01c007559e9080124"
server_key_iv: "b4f5be195faf07059f2cd0ca0026c1c2"
server_key_version: "encryptionkey@0"
I20260810 14:40:37.127055 19797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:37.148087 19797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:37.151628 19797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:37.151829 19797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:37.152478 19797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:37.153417 19797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:37.153469 19797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:37.153532 19797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:37.153558 19797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:37.162741 19797 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:40467
I20260810 14:40:37.162768 19976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:40467 every 8 connection(s)
I20260810 14:40:37.163700 19797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:40:37.172528 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 19797
I20260810 14:40:37.172653 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:40:37.172927 15696 external_mini_cluster.cc:1428] Setting key fa77b000fc7c409a362a5f73c3222b0e
Aug 10 14:40:37 dist-test-slave-rmhg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372837, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:37.178907 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:37.165712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:40625 (local address 127.15.84.62:45123)
0810 14:40:37.165836 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:37.165840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:37.166761 (+ 921us) server_negotiation.cc:407] Connection header received
0810 14:40:37.168100 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:37.168103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:37.168166 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:37.168273 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:37.169880 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:37.170702 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:37.171935 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:37.172159 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:37.172259 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:37.174972 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:37.174976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:37.174979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:37.175008 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:37.176466 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:37.177131 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:37.177134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:37.177135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:37.177271 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:37.177697 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:37.177699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:37.177700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:37.177889 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:37.177966 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:37.178595 (+ 629us) server_negotiation.cc:299] Negotiation successful
0810 14:40:37.178740 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:40:37.179842 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:37.166075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:45123 (local address 127.15.84.2:40625)
0810 14:40:37.166574 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:37.166620 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:37.167867 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:37.168491 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:37.168498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:37.169006 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:37.169679 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:37.169703 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:37.170851 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:37.170854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:37.171821 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:37.171833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:37.172284 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:37.172287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:37.172435 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:37.173105 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:37.173123 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:37.174793 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:37.176609 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:37.176616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:37.176649 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:37.177033 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:37.177357 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:37.177361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:37.177363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:37.177565 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:37.177988 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:37.177995 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:37.178332 (+ 337us) client_negotiation.cc:769] Sending connection context
0810 14:40:37.178528 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 14:40:37.178918 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":125,"threads_started":1}
I20260810 14:40:37.181169 19977 heartbeater.cc:347] Connected to a master server at 127.15.84.62:45123
I20260810 14:40:37.181427 19977 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:37.181964 19977 heartbeater.cc:511] Master 127.15.84.62:45123 requested a full tablet report, sending...
I20260810 14:40:37.183159 19506 ts_manager.cc:194] Registered new tserver with Master: 054ee4a6e1ee417fad851ecb56873ffb (127.15.84.2:40467)
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
I20260810 14:40:37.183992 19506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:40625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:40:37.202225 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:45123
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:37.320395 19982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:37.320698 19982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:37.320744 19982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:37.324920 19982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:37.325004 19982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:37.325064 19982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:40:37.330732 19982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:45123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:37.331954 19982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:37.333344 19982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:37.343973 19994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:37.343973 19988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:37.344058 19982 server_base.cc:1034] running on GCE node
W20260810 14:40:37.343974 19990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:37.344679 19982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:37.345410 19982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:37.346589 19982 hybrid_clock.cc:648] HybridClock initialized: now 1786372837346563 us; error 36 us; skew 500 ppm
Aug 10 14:40:37 dist-test-slave-rmhg krb5kdc[19437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372837, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:37.349578 19982 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:40:37.350704 19982 webserver.cc:460] Webserver started at http://127.15.84.3:39297/ using document root <none> and password file <none>
I20260810 14:40:37.351320 19982 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:37.351388 19982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:37.351603 19982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:37.353888 19982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "c0fbe2d2a0d84e48a467460c947da347"
format_stamp: "Formatted at 2026-08-10 14:40:37 on dist-test-slave-rmhg"
server_key: "77da83281a1d0cbf65a5cb80a1790f00"
server_key_iv: "00aeaecef1069a39c5d4060d238867fc"
server_key_version: "encryptionkey@0"
I20260810 14:40:37.354471 19982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "c0fbe2d2a0d84e48a467460c947da347"
format_stamp: "Formatted at 2026-08-10 14:40:37 on dist-test-slave-rmhg"
server_key: "77da83281a1d0cbf65a5cb80a1790f00"
server_key_iv: "00aeaecef1069a39c5d4060d238867fc"
server_key_version: "encryptionkey@0"
I20260810 14:40:37.358428 19982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 14:40:37.361039 20003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:37.362378 19982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:40:37.362497 19982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "c0fbe2d2a0d84e48a467460c947da347"
format_stamp: "Formatted at 2026-08-10 14:40:37 on dist-test-slave-rmhg"
server_key: "77da83281a1d0cbf65a5cb80a1790f00"
server_key_iv: "00aeaecef1069a39c5d4060d238867fc"
server_key_version: "encryptionkey@0"
I20260810 14:40:37.362615 19982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:37.403113 19982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:37.406196 19982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:37.406399 19982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:37.407025 19982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:37.407917 19982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:37.407979 19982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:37.408023 19982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:37.408039 19982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:37.417233 19982 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:36081
I20260810 14:40:37.417258 20175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:36081 every 8 connection(s)
I20260810 14:40:37.418218 19982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:40:37.420544 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 19982
I20260810 14:40:37.420661 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:40:37.420959 15696 external_mini_cluster.cc:1428] Setting key 5df0a902303726954f8fe1aa8b53252a
Aug 10 14:40:37 dist-test-slave-rmhg krb5kdc[19437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372837, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:37.432554 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:37.420097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:39643 (local address 127.15.84.62:45123)
0810 14:40:37.420315 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:37.420320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:37.421301 (+ 981us) server_negotiation.cc:407] Connection header received
0810 14:40:37.422870 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:37.422874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:37.422938 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:37.423024 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:37.424499 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:37.425034 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:37.426320 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:37.426561 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:37.426646 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:37.429241 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:37.429245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:37.429247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:37.429274 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:37.430454 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:37.431072 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:37.431075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:37.431076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:37.431167 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:37.431482 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:37.431484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:37.431485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:37.431665 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:37.431742 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:37.432313 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 14:40:37.432430 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 14:40:37.433430 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:37.420567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:45123 (local address 127.15.84.3:39643)
0810 14:40:37.421108 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:37.421196 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:37.422640 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:37.423360 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:37.423371 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:37.423804 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:37.424346 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:37.424362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:37.425205 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:37.425210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:37.426180 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:37.426191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:37.426704 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:37.426707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:37.426862 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:37.427438 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:37.427456 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:37.429029 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:37.430603 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:37.430611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:37.430648 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:37.430951 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:37.431256 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:37.431259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:37.431260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:37.431410 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:37.431766 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:37.431773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:37.432094 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 14:40:37.432262 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 14:40:37.432542 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":135,"threads_started":1}
I20260810 14:40:37.434633 20176 heartbeater.cc:347] Connected to a master server at 127.15.84.62:45123
I20260810 14:40:37.434890 20176 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:37.435491 20176 heartbeater.cc:511] Master 127.15.84.62:45123 requested a full tablet report, sending...
I20260810 14:40:37.436478 19506 ts_manager.cc:194] Registered new tserver with Master: c0fbe2d2a0d84e48a467460c947da347 (127.15.84.3:36081)
I20260810 14:40:37.437027 19506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:39643
I20260810 14:40:37.446976 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 14:40:37.466346 19564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:37.464665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.15.84.62:45123)
0810 14:40:37.464832 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:37.464838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:37.464972 (+ 134us) server_negotiation.cc:407] Connection header received
0810 14:40:37.465136 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:37.465138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:37.465239 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:37.465310 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:37.466186 (+ 876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43926: BlockingRecv error: recv got EOF from 127.0.0.1:43926 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:40:37.467711 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 19596
I20260810 14:40:37.486733 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 19797
I20260810 14:40:37.510246 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 19982
I20260810 14:40:37.524503 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 19457
2026-08-10T14:40:37Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2361 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:37 dist-test-slave-rmhg krb5kdc[20196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:37 dist-test-slave-rmhg krb5kdc[20196](info): set up 2 sockets
Aug 10 14:40:37 dist-test-slave-rmhg krb5kdc[20196](info): commencing operation
krb5kdc: starting...
W20260810 14:40:38.677134 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:40:38 dist-test-slave-rmhg krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:40:38.807716 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:37529
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:43831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:37529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:38.925766 20213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:38.926080 20213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:38.926139 20213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:38.930233 20213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:38.930312 20213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:38.930338 20213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:38.930361 20213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:38.930379 20213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:38.935909 20213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:43831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:37529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:37529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:38.937084 20213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:38.938175 20213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:38.947355 20223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:38.947342 20221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:38.947340 20220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:38.947425 20213 server_base.cc:1034] running on GCE node
I20260810 14:40:38.948239 20213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:38.949525 20213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:38.950716 20213 hybrid_clock.cc:648] HybridClock initialized: now 1786372838950695 us; error 43 us; skew 500 ppm
Aug 10 14:40:38 dist-test-slave-rmhg krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372838, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:38.953550 20213 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:40:38.954490 20213 webserver.cc:460] Webserver started at http://127.15.84.62:36309/ using document root <none> and password file <none>
I20260810 14:40:38.955076 20213 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:38.955143 20213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:38.955348 20213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:38.957502 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "2d6c5412d8674b188d9e0bebf7b6ee26"
format_stamp: "Formatted at 2026-08-10 14:40:38 on dist-test-slave-rmhg"
server_key: "aa1ba15bc7f4126b81e7c81532d80f07"
server_key_iv: "e9b67510be902a5002c970a93921402c"
server_key_version: "encryptionkey@0"
I20260810 14:40:38.958052 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "2d6c5412d8674b188d9e0bebf7b6ee26"
format_stamp: "Formatted at 2026-08-10 14:40:38 on dist-test-slave-rmhg"
server_key: "aa1ba15bc7f4126b81e7c81532d80f07"
server_key_iv: "e9b67510be902a5002c970a93921402c"
server_key_version: "encryptionkey@0"
I20260810 14:40:38.961844 20213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:40:38.964329 20234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:38.965651 20213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:40:38.965791 20213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "2d6c5412d8674b188d9e0bebf7b6ee26"
format_stamp: "Formatted at 2026-08-10 14:40:38 on dist-test-slave-rmhg"
server_key: "aa1ba15bc7f4126b81e7c81532d80f07"
server_key_iv: "e9b67510be902a5002c970a93921402c"
server_key_version: "encryptionkey@0"
I20260810 14:40:38.965895 20213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:38.987600 20213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:38.990816 20213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:38.991022 20213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:38.998545 20213 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:37529
I20260810 14:40:38.998814 20306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:37529 every 8 connection(s)
I20260810 14:40:38.999558 20213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:39.002578 20307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:39.005956 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 20213
I20260810 14:40:39.006100 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:39.006394 15696 external_mini_cluster.cc:1428] Setting key 80318b71edde3841abcde23f18f2252d
I20260810 14:40:39.009289 20307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26: Bootstrap starting.
I20260810 14:40:39.012053 20307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:39.013038 20307 log.cc:824] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:39.015621 20307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26: No bootstrap required, opened a new log
Aug 10 14:40:39 dist-test-slave-rmhg krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:39.019043 20307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6c5412d8674b188d9e0bebf7b6ee26" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37529 } }
I20260810 14:40:39.019315 20307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:39.019389 20307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d6c5412d8674b188d9e0bebf7b6ee26, State: Initialized, Role: FOLLOWER
I20260810 14:40:39.019968 20307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6c5412d8674b188d9e0bebf7b6ee26" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37529 } }
I20260810 14:40:39.020128 20307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:39.020200 20307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:39.020305 20307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:39.021656 20307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6c5412d8674b188d9e0bebf7b6ee26" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37529 } }
I20260810 14:40:39.022109 20307 leader_election.cc:302] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d6c5412d8674b188d9e0bebf7b6ee26; no voters:
I20260810 14:40:39.022475 20307 leader_election.cc:288] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:39.022591 20317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:39.022816 20317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [term 1 LEADER]: Becoming Leader. State: Replica: 2d6c5412d8674b188d9e0bebf7b6ee26, State: Running, Role: LEADER
I20260810 14:40:39.023085 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:39.007865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.15.84.62:37529)
0810 14:40:39.008365 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:39.008382 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:39.008423 (+ 41us) server_negotiation.cc:407] Connection header received
0810 14:40:39.009108 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:39.009126 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:39.009573 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:39.009900 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:39.010896 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:39.012010 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:39.013268 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:39.013986 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:39.014134 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:39.016986 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:39.016993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:39.017009 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:39.017054 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:39.019404 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:39.019985 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:39.019991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:39.019996 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:39.020129 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:39.020459 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:39.020463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:39.020465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:39.021040 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:39.021299 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:39.021622 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 14:40:39.021982 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":164,"threads_started":1}
I20260810 14:40:39.023245 20317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6c5412d8674b188d9e0bebf7b6ee26" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37529 } }
I20260810 14:40:39.023808 20307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:39.025673 20319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d6c5412d8674b188d9e0bebf7b6ee26. Latest consensus state: current_term: 1 leader_uuid: "2d6c5412d8674b188d9e0bebf7b6ee26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6c5412d8674b188d9e0bebf7b6ee26" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37529 } } }
I20260810 14:40:39.025815 20319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:39.026168 20318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d6c5412d8674b188d9e0bebf7b6ee26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6c5412d8674b188d9e0bebf7b6ee26" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37529 } } }
I20260810 14:40:39.026559 20318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:39.026718 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:40:39.026916 20336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:39.030177 20336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:39.036201 20336 catalog_manager.cc:1347] Generated new cluster ID: 822605c6e2404e82a987740da0729984
I20260810 14:40:39.036389 20336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:39.054243 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:39.048058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.15.84.62:37529)
0810 14:40:39.048187 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:39.048191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:39.048476 (+ 285us) server_negotiation.cc:407] Connection header received
0810 14:40:39.048618 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:39.048621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:39.048671 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:39.048762 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:39.049419 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:39.050250 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:39.051453 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:39.051894 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:39.051986 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:39.052972 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:39.052976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:39.052979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:39.053722 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:39.053764 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:39.053787 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:39.053878 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:39.054010 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 14:40:39.054084 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:40:39.068503 20336 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:39.069958 20336 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:39.085224 20336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d6c5412d8674b188d9e0bebf7b6ee26: Generated new TSK 0
I20260810 14:40:39.086067 20336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:40:39.091569 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 20213
2026-08-10T14:40:39Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1557 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:39 dist-test-slave-rmhg krb5kdc[20359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:39 dist-test-slave-rmhg krb5kdc[20359](info): set up 2 sockets
Aug 10 14:40:39 dist-test-slave-rmhg krb5kdc[20359](info): commencing operation
krb5kdc: starting...
W20260810 14:40:40.264032 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:40:40 dist-test-slave-rmhg krb5kdc[20359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:40:40.392349 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:39089
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:39089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:40.509202 20375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:40.509521 20375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:40.509582 20375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:40.513764 20375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:40.513854 20375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:40.513881 20375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:40.513906 20375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:40.513924 20375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:40.520750 20375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:39089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:39089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:40.522694 20375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:40.524266 20375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:40.534775 20384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:40.534780 20383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:40.534775 20386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:40.535176 20375 server_base.cc:1034] running on GCE node
I20260810 14:40:40.535785 20375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:40.536966 20375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:40.538120 20375 hybrid_clock.cc:648] HybridClock initialized: now 1786372840538100 us; error 67 us; skew 500 ppm
Aug 10 14:40:40 dist-test-slave-rmhg krb5kdc[20359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372840, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:40.546216 20375 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:40:40.547375 20375 webserver.cc:460] Webserver started at http://127.15.84.62:33829/ using document root <none> and password file <none>
I20260810 14:40:40.548060 20375 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:40.548161 20375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:40.548456 20375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:40.550935 20375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "5bdfdd4d4a024ec393636a7f306e8759"
format_stamp: "Formatted at 2026-08-10 14:40:40 on dist-test-slave-rmhg"
server_key: "8ef6afd03346b4aabbf726dda39b7fff"
server_key_iv: "54cc177704b172631658de434ecc19b3"
server_key_version: "encryptionkey@0"
I20260810 14:40:40.551561 20375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "5bdfdd4d4a024ec393636a7f306e8759"
format_stamp: "Formatted at 2026-08-10 14:40:40 on dist-test-slave-rmhg"
server_key: "8ef6afd03346b4aabbf726dda39b7fff"
server_key_iv: "54cc177704b172631658de434ecc19b3"
server_key_version: "encryptionkey@0"
I20260810 14:40:40.555804 20375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:40:40.558506 20398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:40.559854 20375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:40:40.559988 20375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "5bdfdd4d4a024ec393636a7f306e8759"
format_stamp: "Formatted at 2026-08-10 14:40:40 on dist-test-slave-rmhg"
server_key: "8ef6afd03346b4aabbf726dda39b7fff"
server_key_iv: "54cc177704b172631658de434ecc19b3"
server_key_version: "encryptionkey@0"
I20260810 14:40:40.560101 20375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:40.570168 20375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:40.573140 20375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:40.573364 20375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:40.580590 20375 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:39089
I20260810 14:40:40.580876 20480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:39089 every 8 connection(s)
I20260810 14:40:40.581712 20375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:40.584830 20484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:40.589740 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 20375
I20260810 14:40:40.589877 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:40.590145 15696 external_mini_cluster.cc:1428] Setting key a4dc85fa196c9e8091dd0cf789b155d5
I20260810 14:40:40.591421 20484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759: Bootstrap starting.
I20260810 14:40:40.594302 20484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:40.595235 20484 log.cc:824] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:40.597463 20484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759: No bootstrap required, opened a new log
Aug 10 14:40:40 dist-test-slave-rmhg krb5kdc[20359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:40.601624 20484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdfdd4d4a024ec393636a7f306e8759" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39089 } }
I20260810 14:40:40.601946 20484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:40.602020 20484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bdfdd4d4a024ec393636a7f306e8759, State: Initialized, Role: FOLLOWER
I20260810 14:40:40.602559 20484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdfdd4d4a024ec393636a7f306e8759" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39089 } }
I20260810 14:40:40.602723 20484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:40.602799 20484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:40.602905 20484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:40.604070 20484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdfdd4d4a024ec393636a7f306e8759" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39089 } }
I20260810 14:40:40.604492 20484 leader_election.cc:302] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bdfdd4d4a024ec393636a7f306e8759; no voters:
I20260810 14:40:40.604890 20484 leader_election.cc:288] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:40.605019 20489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:40.605262 20489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [term 1 LEADER]: Becoming Leader. State: Replica: 5bdfdd4d4a024ec393636a7f306e8759, State: Running, Role: LEADER
I20260810 14:40:40.605623 20489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdfdd4d4a024ec393636a7f306e8759" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39089 } }
I20260810 14:40:40.606078 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:40.591811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.15.84.62:39089)
0810 14:40:40.592306 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:40.592325 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:40.592368 (+ 43us) server_negotiation.cc:407] Connection header received
0810 14:40:40.593273 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:40.593299 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:40.593752 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:40.594122 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:40.594956 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:40.596297 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:40.597248 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:40.597946 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:40.598116 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:40.600543 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:40.600550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:40.600569 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:40.600604 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:40.602629 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:40.603181 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:40.603188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:40.603195 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:40.603337 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:40.603572 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:40.603575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:40.603577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:40.604142 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:40.604338 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:40.604645 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 14:40:40.604995 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":150,"threads_started":1}
I20260810 14:40:40.606289 20484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:40.607861 20492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bdfdd4d4a024ec393636a7f306e8759. Latest consensus state: current_term: 1 leader_uuid: "5bdfdd4d4a024ec393636a7f306e8759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdfdd4d4a024ec393636a7f306e8759" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39089 } } }
I20260810 14:40:40.607987 20492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:40.609300 20491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bdfdd4d4a024ec393636a7f306e8759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdfdd4d4a024ec393636a7f306e8759" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39089 } } }
I20260810 14:40:40.609402 20491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759 [sys.catalog]: This master's current role is: LEADER
W20260810 14:40:40.611572 20510 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:40:40.611649 20510 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:40:40.611706 20508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:40.612448 20508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:40.617951 20508 catalog_manager.cc:1347] Generated new cluster ID: 165ccb122409470ebc6294ecf234060b
I20260810 14:40:40.618018 20508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:40.628082 20508 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:40.629496 20508 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:40.646119 20508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bdfdd4d4a024ec393636a7f306e8759: Generated new TSK 0
I20260810 14:40:40.646935 20508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:40:40.685828 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39089
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:40.808475 20517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:40.808768 20517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:40.808820 20517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:40.813064 20517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:40.813140 20517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:40.813247 20517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:40:40.818848 20517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:40.819969 20517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:40.821012 20517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:40.831257 20525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:40.831295 20528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:40.831331 20526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:40.831393 20517 server_base.cc:1034] running on GCE node
I20260810 14:40:40.831985 20517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:40.832656 20517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:40.833827 20517 hybrid_clock.cc:648] HybridClock initialized: now 1786372840833790 us; error 49 us; skew 500 ppm
Aug 10 14:40:40 dist-test-slave-rmhg krb5kdc[20359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372840, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:40.836685 20517 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:40:40.837723 20517 webserver.cc:460] Webserver started at http://127.15.84.1:39859/ using document root <none> and password file <none>
I20260810 14:40:40.838307 20517 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:40.838382 20517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:40.838608 20517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:40.840780 20517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "e539b5cdaa8c4a45902744ace6bde9f4"
format_stamp: "Formatted at 2026-08-10 14:40:40 on dist-test-slave-rmhg"
server_key: "ad2aabc72fb2469e22cf1a1d536e432f"
server_key_iv: "de0852eb360f0e173c3674fc28a70c73"
server_key_version: "encryptionkey@0"
I20260810 14:40:40.841388 20517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "e539b5cdaa8c4a45902744ace6bde9f4"
format_stamp: "Formatted at 2026-08-10 14:40:40 on dist-test-slave-rmhg"
server_key: "ad2aabc72fb2469e22cf1a1d536e432f"
server_key_iv: "de0852eb360f0e173c3674fc28a70c73"
server_key_version: "encryptionkey@0"
I20260810 14:40:40.845366 20517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:40:40.847970 20541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:40.849375 20517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:40.849512 20517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "e539b5cdaa8c4a45902744ace6bde9f4"
format_stamp: "Formatted at 2026-08-10 14:40:40 on dist-test-slave-rmhg"
server_key: "ad2aabc72fb2469e22cf1a1d536e432f"
server_key_iv: "de0852eb360f0e173c3674fc28a70c73"
server_key_version: "encryptionkey@0"
I20260810 14:40:40.849627 20517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:40.861224 20517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:40.864749 20517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:40.864948 20517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:40.865682 20517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:40.866611 20517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:40.866662 20517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:40.866726 20517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:40.866753 20517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:40.875854 20517 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:40971
I20260810 14:40:40.875887 20702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:40971 every 8 connection(s)
I20260810 14:40:40.876803 20517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:40:40.883997 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 20517
I20260810 14:40:40.884127 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:40:40.884454 15696 external_mini_cluster.cc:1428] Setting key 870081ed05986cb408e5303779446905
Aug 10 14:40:40 dist-test-slave-rmhg krb5kdc[20359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372840, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:40.891386 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:40.878684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:55349 (local address 127.15.84.62:39089)
0810 14:40:40.878817 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:40.878820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:40.879676 (+ 856us) server_negotiation.cc:407] Connection header received
0810 14:40:40.880990 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:40.880994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:40.881045 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:40.881120 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:40.882443 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:40.882990 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:40.884266 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:40.884626 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:40.884751 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:40.887669 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:40.887673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:40.887675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:40.887703 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:40.889224 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:40.889758 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:40.889762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:40.889763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:40.889860 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:40.890168 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:40.890171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:40.890173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:40.890408 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:40.890506 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:40.891051 (+ 545us) server_negotiation.cc:299] Negotiation successful
0810 14:40:40.891206 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:40:40.892227 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:40.879005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:39089 (local address 127.15.84.1:55349)
0810 14:40:40.879497 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:40.879544 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:40.880787 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:40.881343 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:40.881351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:40.881763 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:40.882305 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:40.882320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:40.883123 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:40.883126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:40.884116 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:40.884129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:40.884794 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:40.884797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:40.884992 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:40.885675 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:40.885694 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:40.887436 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:40.889347 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:40.889353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:40.889368 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:40.889648 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:40.889939 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:40.889941 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:40.889942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:40.890084 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:40.890494 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:40.890499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:40.890791 (+ 292us) client_negotiation.cc:769] Sending connection context
0810 14:40:40.890958 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 14:40:40.891250 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":127,"threads_started":1}
I20260810 14:40:40.893563 20703 heartbeater.cc:347] Connected to a master server at 127.15.84.62:39089
I20260810 14:40:40.893880 20703 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:40.894549 20703 heartbeater.cc:511] Master 127.15.84.62:39089 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:40:40.896358 20422 ts_manager.cc:194] Registered new tserver with Master: e539b5cdaa8c4a45902744ace6bde9f4 (127.15.84.1:40971)
I20260810 14:40:40.897593 20422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:55349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:40:40.914009 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39089
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:41.032070 20710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:41.032375 20710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:41.032423 20710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:41.036927 20710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:41.037035 20710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:41.037099 20710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:40:41.042829 20710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:41.043972 20710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:41.045035 20710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:41.055553 20720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:41.055588 20718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:41.055553 20717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:41.055850 20710 server_base.cc:1034] running on GCE node
I20260810 14:40:41.056413 20710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:41.057093 20710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:41.058290 20710 hybrid_clock.cc:648] HybridClock initialized: now 1786372841058266 us; error 36 us; skew 500 ppm
Aug 10 14:40:41 dist-test-slave-rmhg krb5kdc[20359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372841, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:41.061259 20710 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:40:41.062282 20710 webserver.cc:460] Webserver started at http://127.15.84.2:34541/ using document root <none> and password file <none>
I20260810 14:40:41.062877 20710 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:41.062944 20710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:41.063165 20710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:41.065492 20710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "919625ffa45a4b11a33baf255120b916"
format_stamp: "Formatted at 2026-08-10 14:40:41 on dist-test-slave-rmhg"
server_key: "3c0d8df0710b36118c085e8d651f216e"
server_key_iv: "2ca0811a5aa9cce26d0b4d685b2aef28"
server_key_version: "encryptionkey@0"
I20260810 14:40:41.066094 20710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "919625ffa45a4b11a33baf255120b916"
format_stamp: "Formatted at 2026-08-10 14:40:41 on dist-test-slave-rmhg"
server_key: "3c0d8df0710b36118c085e8d651f216e"
server_key_iv: "2ca0811a5aa9cce26d0b4d685b2aef28"
server_key_version: "encryptionkey@0"
I20260810 14:40:41.070258 20710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:40:41.072937 20731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:41.074349 20710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:40:41.074498 20710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "919625ffa45a4b11a33baf255120b916"
format_stamp: "Formatted at 2026-08-10 14:40:41 on dist-test-slave-rmhg"
server_key: "3c0d8df0710b36118c085e8d651f216e"
server_key_iv: "2ca0811a5aa9cce26d0b4d685b2aef28"
server_key_version: "encryptionkey@0"
I20260810 14:40:41.074615 20710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:41.101194 20710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:41.104571 20710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:41.104794 20710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:41.105511 20710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:41.106503 20710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:41.106559 20710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:41.106616 20710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:41.106654 20710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:41.116173 20710 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:39535
I20260810 14:40:41.116212 20891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:39535 every 8 connection(s)
I20260810 14:40:41.117245 20710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:40:41.121615 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 20710
I20260810 14:40:41.121750 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:40:41.122047 15696 external_mini_cluster.cc:1428] Setting key 1627a7da5b211c3ba62274a74f350b44
Aug 10 14:40:41 dist-test-slave-rmhg krb5kdc[20359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372841, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
I20260810 14:40:41.132939 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:41.119232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:58769 (local address 127.15.84.62:39089)
0810 14:40:41.119385 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:41.119389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:41.120454 (+ 1065us) server_negotiation.cc:407] Connection header received
0810 14:40:41.121975 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:41.121980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:41.122046 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:41.122142 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:41.123879 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:41.124609 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:41.126109 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:41.126350 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:41.126429 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:41.129369 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:41.129372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:41.129374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:41.129404 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:41.130794 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:41.131331 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:41.131334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:41.131335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:41.131419 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:41.131772 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:41.131775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:41.131776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:41.131977 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:41.132068 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:41.132629 (+ 561us) server_negotiation.cc:299] Negotiation successful
0810 14:40:41.132769 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:40:41.133946 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:41.119690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:39089 (local address 127.15.84.2:58769)
0810 14:40:41.120279 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:41.120334 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:41.121707 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:41.122365 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:41.122374 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:41.122886 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:41.123707 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:41.123732 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:41.124774 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:41.124777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:41.125936 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:41.125949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:41.126512 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:41.126516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:41.126739 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:41.127409 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:41.127431 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:41.129132 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:41.130918 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:41.130926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:41.130945 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:41.131219 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:41.131504 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:41.131507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:41.131509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:41.131670 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:41.132082 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:41.132089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:41.132376 (+ 287us) client_negotiation.cc:769] Sending connection context
0810 14:40:41.132550 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 14:40:41.132930 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":148,"threads_started":1}
I20260810 14:40:41.135285 20892 heartbeater.cc:347] Connected to a master server at 127.15.84.62:39089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
I20260810 14:40:41.135560 20892 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:41.136152 20892 heartbeater.cc:511] Master 127.15.84.62:39089 requested a full tablet report, sending...
I20260810 14:40:41.137480 20422 ts_manager.cc:194] Registered new tserver with Master: 919625ffa45a4b11a33baf255120b916 (127.15.84.2:39535)
I20260810 14:40:41.138267 20422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:58769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:40:41.152917 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39089
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:41.270862 20898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:41.271155 20898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:41.271199 20898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:41.275358 20898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:41.275439 20898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:41.275501 20898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:40:41.281129 20898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:41.282297 20898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:41.283308 20898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:41.293608 20903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:41.293601 20906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:41.293720 20898 server_base.cc:1034] running on GCE node
W20260810 14:40:41.293601 20909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:41.294366 20898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:41.295094 20898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:41.296263 20898 hybrid_clock.cc:648] HybridClock initialized: now 1786372841296240 us; error 37 us; skew 500 ppm
Aug 10 14:40:41 dist-test-slave-rmhg krb5kdc[20359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372841, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:41.299064 20898 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:40:41.300122 20898 webserver.cc:460] Webserver started at http://127.15.84.3:36957/ using document root <none> and password file <none>
I20260810 14:40:41.300715 20898 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:41.300791 20898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:41.301010 20898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:41.303242 20898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "feab7e35077f4c37b0e2f2fe8d7b60e8"
format_stamp: "Formatted at 2026-08-10 14:40:41 on dist-test-slave-rmhg"
server_key: "7f97ef531a4679e106e47a7d859e614f"
server_key_iv: "99b493ea570420cc291473e0c973d3e8"
server_key_version: "encryptionkey@0"
I20260810 14:40:41.303831 20898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "feab7e35077f4c37b0e2f2fe8d7b60e8"
format_stamp: "Formatted at 2026-08-10 14:40:41 on dist-test-slave-rmhg"
server_key: "7f97ef531a4679e106e47a7d859e614f"
server_key_iv: "99b493ea570420cc291473e0c973d3e8"
server_key_version: "encryptionkey@0"
I20260810 14:40:41.308044 20898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:40:41.310736 20918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:41.312062 20898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:40:41.312196 20898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "feab7e35077f4c37b0e2f2fe8d7b60e8"
format_stamp: "Formatted at 2026-08-10 14:40:41 on dist-test-slave-rmhg"
server_key: "7f97ef531a4679e106e47a7d859e614f"
server_key_iv: "99b493ea570420cc291473e0c973d3e8"
server_key_version: "encryptionkey@0"
I20260810 14:40:41.312309 20898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:41.332281 20898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:41.335466 20898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:41.335659 20898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:41.336295 20898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:41.337221 20898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:41.337273 20898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:41.337328 20898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:41.337352 20898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:41.346535 20898 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:34701
I20260810 14:40:41.346570 21084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:34701 every 8 connection(s)
I20260810 14:40:41.347517 20898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:40:41.351022 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 20898
I20260810 14:40:41.351167 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:40:41.351488 15696 external_mini_cluster.cc:1428] Setting key 55bdc579306c53cb2cce5057afb44b65
Aug 10 14:40:41 dist-test-slave-rmhg krb5kdc[20359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372841, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:41.362185 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:41.349789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:55503 (local address 127.15.84.62:39089)
0810 14:40:41.349930 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:41.349934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:41.350840 (+ 906us) server_negotiation.cc:407] Connection header received
0810 14:40:41.352609 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:41.352612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:41.352663 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:41.352735 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:41.354394 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:41.354942 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:41.356074 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:41.356280 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:41.356364 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:41.358912 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:41.358915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:41.358917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:41.358947 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:41.360228 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:41.360723 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:41.360726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:41.360728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:41.360802 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:41.361067 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:41.361069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:41.361070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:41.361317 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:41.361396 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:41.361940 (+ 544us) server_negotiation.cc:299] Negotiation successful
0810 14:40:41.362049 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:40:41.363111 21090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:41.350147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:39089 (local address 127.15.84.3:55503)
0810 14:40:41.350651 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:41.350700 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:41.352374 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:41.352915 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:41.352926 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:41.353497 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:41.354221 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:41.354245 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:41.355039 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:41.355041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:41.355979 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:41.355989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:41.356403 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:41.356406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:41.356552 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:41.357183 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:41.357207 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:41.358729 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:41.360340 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:41.360347 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:41.360362 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:41.360640 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:41.360863 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:41.360866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:41.360867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:41.361003 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:41.361424 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:41.361430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:41.361721 (+ 291us) client_negotiation.cc:769] Sending connection context
0810 14:40:41.361906 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 14:40:41.362210 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":115,"threads_started":1}
I20260810 14:40:41.364364 21085 heartbeater.cc:347] Connected to a master server at 127.15.84.62:39089
I20260810 14:40:41.364670 21085 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:41.365357 21085 heartbeater.cc:511] Master 127.15.84.62:39089 requested a full tablet report, sending...
I20260810 14:40:41.366454 20422 ts_manager.cc:194] Registered new tserver with Master: feab7e35077f4c37b0e2f2fe8d7b60e8 (127.15.84.3:34701)
I20260810 14:40:41.367043 20422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:55503
I20260810 14:40:41.378533 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 14:40:41 dist-test-slave-rmhg krb5kdc[20359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372841, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 14:40:41 dist-test-slave-rmhg krb5kdc[20359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372841, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:41.409793 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:41.399790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.15.84.62:39089)
0810 14:40:41.399955 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:41.399960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:41.400146 (+ 186us) server_negotiation.cc:407] Connection header received
0810 14:40:41.400327 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:41.400330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:41.400376 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:41.400455 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:41.401386 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:41.401916 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:41.403134 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:41.403347 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:41.403417 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:41.406341 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:41.406347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:41.406350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:41.406387 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:41.408025 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:41.408536 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:41.408539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:41.408540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:41.408626 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:41.408878 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:41.408880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:41.408881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:41.409141 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:41.409304 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:41.409544 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 14:40:41.409655 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 14:40:41.410251 20422 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49450
I20260810 14:40:41.411284 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 20517
I20260810 14:40:41.433979 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 20710
I20260810 14:40:41.444186 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 20898
I20260810 14:40:41.460624 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 20375
2026-08-10T14:40:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2372 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:41 dist-test-slave-rmhg krb5kdc[21102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:41 dist-test-slave-rmhg krb5kdc[21102](info): set up 2 sockets
Aug 10 14:40:41 dist-test-slave-rmhg krb5kdc[21102](info): commencing operation
krb5kdc: starting...
W20260810 14:40:42.636472 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:40:42 dist-test-slave-rmhg krb5kdc[21102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:40:42.766497 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:46779
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:46779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:42.885061 21120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:42.885409 21120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:42.885469 21120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:42.889599 21120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:42.889671 21120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:42.889698 21120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:42.889716 21120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:42.889739 21120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:42.895293 21120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:46779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:46779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:42.896394 21120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:42.897468 21120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:42.906256 21127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:42.906365 21129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:42.906311 21120 server_base.cc:1034] running on GCE node
W20260810 14:40:42.906303 21126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:42.907047 21120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:42.908291 21120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:42.909466 21120 hybrid_clock.cc:648] HybridClock initialized: now 1786372842909446 us; error 38 us; skew 500 ppm
Aug 10 14:40:42 dist-test-slave-rmhg krb5kdc[21102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372842, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:42.912250 21120 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:40:42.913285 21120 webserver.cc:460] Webserver started at http://127.15.84.62:43603/ using document root <none> and password file <none>
I20260810 14:40:42.913868 21120 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:42.913940 21120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:42.914184 21120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:42.916373 21120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "c53065c2d29348be9f0cd81a0125a9a2"
format_stamp: "Formatted at 2026-08-10 14:40:42 on dist-test-slave-rmhg"
server_key: "ce97bab211c42c14c8e9e60f38c4318a"
server_key_iv: "45a442e2f1306a645a721228a1ba33ee"
server_key_version: "encryptionkey@0"
I20260810 14:40:42.917032 21120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "c53065c2d29348be9f0cd81a0125a9a2"
format_stamp: "Formatted at 2026-08-10 14:40:42 on dist-test-slave-rmhg"
server_key: "ce97bab211c42c14c8e9e60f38c4318a"
server_key_iv: "45a442e2f1306a645a721228a1ba33ee"
server_key_version: "encryptionkey@0"
I20260810 14:40:42.920954 21120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 14:40:42.923548 21141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:42.925022 21120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:40:42.925211 21120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "c53065c2d29348be9f0cd81a0125a9a2"
format_stamp: "Formatted at 2026-08-10 14:40:42 on dist-test-slave-rmhg"
server_key: "ce97bab211c42c14c8e9e60f38c4318a"
server_key_iv: "45a442e2f1306a645a721228a1ba33ee"
server_key_version: "encryptionkey@0"
I20260810 14:40:42.925336 21120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:42.935564 21120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:42.938728 21120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:42.938922 21120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:42.946334 21120 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:46779
I20260810 14:40:42.946353 21209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:46779 every 8 connection(s)
I20260810 14:40:42.947348 21120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:42.950424 21210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:42.954373 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 21120
I20260810 14:40:42.954520 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:42.954838 15696 external_mini_cluster.cc:1428] Setting key e4bd90983bee063ee2c3cc2512ee1ba0
I20260810 14:40:42.957271 21210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2: Bootstrap starting.
I20260810 14:40:42.959656 21210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:42.960471 21210 log.cc:824] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:42.962647 21210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2: No bootstrap required, opened a new log
Aug 10 14:40:42 dist-test-slave-rmhg krb5kdc[21102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:42.966079 21210 raft_consensus.cc:348] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53065c2d29348be9f0cd81a0125a9a2" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46779 } }
I20260810 14:40:42.966403 21210 raft_consensus.cc:374] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:42.966482 21210 raft_consensus.cc:729] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c53065c2d29348be9f0cd81a0125a9a2, State: Initialized, Role: FOLLOWER
I20260810 14:40:42.966989 21210 consensus_queue.cc:260] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53065c2d29348be9f0cd81a0125a9a2" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46779 } }
I20260810 14:40:42.967150 21210 raft_consensus.cc:388] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:42.967222 21210 raft_consensus.cc:482] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:42.967329 21210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:42.968267 21210 raft_consensus.cc:504] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53065c2d29348be9f0cd81a0125a9a2" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46779 } }
I20260810 14:40:42.968662 21210 leader_election.cc:302] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c53065c2d29348be9f0cd81a0125a9a2; no voters:
I20260810 14:40:42.969103 21210 leader_election.cc:288] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:42.969429 21220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:42.969703 21220 raft_consensus.cc:686] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [term 1 LEADER]: Becoming Leader. State: Replica: c53065c2d29348be9f0cd81a0125a9a2, State: Running, Role: LEADER
I20260810 14:40:42.970085 21220 consensus_queue.cc:237] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53065c2d29348be9f0cd81a0125a9a2" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46779 } }
I20260810 14:40:42.970395 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:42.956505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.15.84.62:46779)
0810 14:40:42.957010 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:42.957026 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:42.957065 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:40:42.957853 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:42.957876 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:42.958242 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:42.958571 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:42.959454 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:42.960558 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:42.961510 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:42.962043 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:42.962207 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:42.964786 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:42.964792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:42.964808 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:42.964840 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:42.967015 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:42.967517 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:42.967523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:42.967528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:42.967650 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:42.967935 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:42.967938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:42.967939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:42.968374 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:42.968549 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:42.968915 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 14:40:42.969259 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":157,"threads_started":1}
I20260810 14:40:42.970564 21210 sys_catalog.cc:564] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:42.972353 21223 sys_catalog.cc:455] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c53065c2d29348be9f0cd81a0125a9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53065c2d29348be9f0cd81a0125a9a2" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46779 } } }
I20260810 14:40:42.972390 21225 sys_catalog.cc:455] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c53065c2d29348be9f0cd81a0125a9a2. Latest consensus state: current_term: 1 leader_uuid: "c53065c2d29348be9f0cd81a0125a9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53065c2d29348be9f0cd81a0125a9a2" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46779 } } }
I20260810 14:40:42.972484 21223 sys_catalog.cc:458] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:42.972488 21225 sys_catalog.cc:458] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:42.973343 21236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:42.976603 21236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:42.982999 21236 catalog_manager.cc:1347] Generated new cluster ID: afba48d915f94a33beaa934e5be16fc4
I20260810 14:40:42.983079 21236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:43.002676 21236 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:43.003787 21236 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:43.010525 21236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c53065c2d29348be9f0cd81a0125a9a2: Generated new TSK 0
I20260810 14:40:43.011366 21236 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:40:43.048254 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:46779
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:43.174847 21252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:43.175158 21252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:43.175206 21252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:43.179495 21252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:43.179589 21252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:43.179652 21252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:40:43.185322 21252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:46779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:43.186539 21252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:43.187600 21252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:43.198299 21262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:43.198300 21260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:43.198318 21259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:43.198539 21252 server_base.cc:1034] running on GCE node
I20260810 14:40:43.199064 21252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:43.199775 21252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:43.200953 21252 hybrid_clock.cc:648] HybridClock initialized: now 1786372843200925 us; error 43 us; skew 500 ppm
Aug 10 14:40:43 dist-test-slave-rmhg krb5kdc[21102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372843, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:43.203961 21252 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:40:43.204958 21252 webserver.cc:460] Webserver started at http://127.15.84.1:44791/ using document root <none> and password file <none>
I20260810 14:40:43.205585 21252 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:43.205659 21252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:43.205889 21252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:43.208086 21252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "2a3ee0c794774698b1bdec0962ca75d1"
format_stamp: "Formatted at 2026-08-10 14:40:43 on dist-test-slave-rmhg"
server_key: "899a48809d21fda01ff71a09bcabedcd"
server_key_iv: "e0a18f503cfd461482a5da3c3745e962"
server_key_version: "encryptionkey@0"
I20260810 14:40:43.208652 21252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "2a3ee0c794774698b1bdec0962ca75d1"
format_stamp: "Formatted at 2026-08-10 14:40:43 on dist-test-slave-rmhg"
server_key: "899a48809d21fda01ff71a09bcabedcd"
server_key_iv: "e0a18f503cfd461482a5da3c3745e962"
server_key_version: "encryptionkey@0"
I20260810 14:40:43.212687 21252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:40:43.215376 21272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:43.216676 21252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:40:43.216799 21252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "2a3ee0c794774698b1bdec0962ca75d1"
format_stamp: "Formatted at 2026-08-10 14:40:43 on dist-test-slave-rmhg"
server_key: "899a48809d21fda01ff71a09bcabedcd"
server_key_iv: "e0a18f503cfd461482a5da3c3745e962"
server_key_version: "encryptionkey@0"
I20260810 14:40:43.216904 21252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:43.226385 21252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:43.229552 21252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:43.229770 21252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:43.230381 21252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:43.231249 21252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:43.231299 21252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:43.231354 21252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:43.231379 21252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:43.240250 21252 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:40501
I20260810 14:40:43.240275 21434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:40501 every 8 connection(s)
I20260810 14:40:43.241230 21252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:40:43.246811 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 21252
I20260810 14:40:43.246930 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:40:43.247200 15696 external_mini_cluster.cc:1428] Setting key a3b062aab70bd78a35dd30239681c7e7
Aug 10 14:40:43 dist-test-slave-rmhg krb5kdc[21102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372843, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:43.255744 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:43.243130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:54885 (local address 127.15.84.62:46779)
0810 14:40:43.243262 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:43.243266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:43.244193 (+ 927us) server_negotiation.cc:407] Connection header received
0810 14:40:43.245597 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:43.245600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:43.245651 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:43.245727 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:43.247006 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.247587 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.248973 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.249245 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.249332 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:43.251891 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:43.251895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:43.251897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:43.251926 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:43.253487 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:43.254056 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:43.254060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:43.254062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:43.254163 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:43.254500 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:43.254504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:43.254506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:43.254776 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:43.254893 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:43.255400 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 14:40:43.255544 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:40:43.256608 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:43.243527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:46779 (local address 127.15.84.1:54885)
0810 14:40:43.244020 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:43.244066 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:43.245392 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:43.245894 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:43.245900 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:43.246295 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:43.246831 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:43.246845 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.247734 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.247737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:43.248848 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:43.248859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.249398 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.249402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:43.249585 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:43.250146 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:43.250163 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:43.251738 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:43.253618 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:43.253627 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:43.253645 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:43.253953 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:43.254248 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:43.254250 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:43.254252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:43.254398 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:43.254890 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:43.254895 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:43.255153 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 14:40:43.255324 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 14:40:43.255603 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:40:43.257970 21435 heartbeater.cc:347] Connected to a master server at 127.15.84.62:46779
I20260810 14:40:43.258230 21435 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:43.258754 21435 heartbeater.cc:511] Master 127.15.84.62:46779 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
I20260810 14:40:43.260643 21163 ts_manager.cc:194] Registered new tserver with Master: 2a3ee0c794774698b1bdec0962ca75d1 (127.15.84.1:40501)
I20260810 14:40:43.261998 21163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:54885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:40:43.276378 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:46779
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:43.393618 21447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:43.393958 21447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:43.394033 21447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:43.398340 21447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:43.398417 21447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:43.398481 21447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:40:43.403947 21447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:46779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:43.405045 21447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:43.406111 21447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:43.415942 21456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:43.415946 21453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:43.416164 21447 server_base.cc:1034] running on GCE node
W20260810 14:40:43.416280 21454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:43.416677 21447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:43.417372 21447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:43.418538 21447 hybrid_clock.cc:648] HybridClock initialized: now 1786372843418513 us; error 32 us; skew 500 ppm
Aug 10 14:40:43 dist-test-slave-rmhg krb5kdc[21102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372843, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:43.421588 21447 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:40:43.422586 21447 webserver.cc:460] Webserver started at http://127.15.84.2:37085/ using document root <none> and password file <none>
I20260810 14:40:43.423211 21447 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:43.423283 21447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:43.423497 21447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:43.425781 21447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "44c0e9c0acb74c6697ba91f9e8e6d450"
format_stamp: "Formatted at 2026-08-10 14:40:43 on dist-test-slave-rmhg"
server_key: "9d87b2cd631883b0372c6e9d8ca5ce26"
server_key_iv: "cf24a74f57b13a2cd4de89ee3a8ac2d9"
server_key_version: "encryptionkey@0"
I20260810 14:40:43.426357 21447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "44c0e9c0acb74c6697ba91f9e8e6d450"
format_stamp: "Formatted at 2026-08-10 14:40:43 on dist-test-slave-rmhg"
server_key: "9d87b2cd631883b0372c6e9d8ca5ce26"
server_key_iv: "cf24a74f57b13a2cd4de89ee3a8ac2d9"
server_key_version: "encryptionkey@0"
I20260810 14:40:43.430394 21447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 14:40:43.433012 21466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:43.434353 21447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:40:43.434477 21447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "44c0e9c0acb74c6697ba91f9e8e6d450"
format_stamp: "Formatted at 2026-08-10 14:40:43 on dist-test-slave-rmhg"
server_key: "9d87b2cd631883b0372c6e9d8ca5ce26"
server_key_iv: "cf24a74f57b13a2cd4de89ee3a8ac2d9"
server_key_version: "encryptionkey@0"
I20260810 14:40:43.434584 21447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:43.464924 21447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:43.467932 21447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:43.468130 21447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:43.468770 21447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:43.469741 21447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:43.469794 21447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:43.469847 21447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:43.469873 21447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:43.478868 21447 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:44865
I20260810 14:40:43.478904 21623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:44865 every 8 connection(s)
I20260810 14:40:43.479825 21447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:40:43.484413 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 21447
I20260810 14:40:43.484546 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:40:43.484870 15696 external_mini_cluster.cc:1428] Setting key b7ad98e74932a99a1d0644b7a68fe40c
Aug 10 14:40:43 dist-test-slave-rmhg krb5kdc[21102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372843, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:43.494792 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:43.481614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:36889 (local address 127.15.84.62:46779)
0810 14:40:43.481785 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:43.481791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:43.482713 (+ 922us) server_negotiation.cc:407] Connection header received
0810 14:40:43.484074 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:43.484077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:43.484126 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:43.484205 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:43.485853 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.486469 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.488063 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.488282 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.488350 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:43.491046 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:43.491050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:43.491052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:43.491078 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:43.492518 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:43.493145 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:43.493167 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:43.493168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:43.493251 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:43.493619 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:43.493621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:43.493622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:43.493818 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:43.493889 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:43.494539 (+ 650us) server_negotiation.cc:299] Negotiation successful
0810 14:40:43.494649 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
I20260810 14:40:43.496054 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:43.482053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:46779 (local address 127.15.84.2:36889)
0810 14:40:43.482545 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:43.482591 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:43.483866 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:43.484457 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:43.484482 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:43.485017 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:43.485712 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:43.485734 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.486602 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.486606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:43.487953 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:43.487966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.488424 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.488427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:43.488653 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:43.489274 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:43.489291 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:43.490855 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:43.492651 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:43.492659 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:43.492679 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:43.493049 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:43.493330 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:43.493333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:43.493336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:43.493531 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:43.493911 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:43.493918 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:43.494291 (+ 373us) client_negotiation.cc:769] Sending connection context
0810 14:40:43.494526 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 14:40:43.494913 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":117,"threads_started":1}
I20260810 14:40:43.497277 21624 heartbeater.cc:347] Connected to a master server at 127.15.84.62:46779
I20260810 14:40:43.497514 21624 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:43.498013 21624 heartbeater.cc:511] Master 127.15.84.62:46779 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
I20260810 14:40:43.499099 21163 ts_manager.cc:194] Registered new tserver with Master: 44c0e9c0acb74c6697ba91f9e8e6d450 (127.15.84.2:44865)
I20260810 14:40:43.499704 21163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:36889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:40:43.513892 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:46779
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:43.632249 21634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:43.632551 21634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:43.632601 21634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:43.636881 21634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:43.636978 21634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:43.637061 21634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:40:43.642903 21634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:46779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:43.644131 21634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:43.645280 21634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:43.655324 21640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:43.655405 21634 server_base.cc:1034] running on GCE node
W20260810 14:40:43.655328 21641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:43.655328 21644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:43.656038 21634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:43.656698 21634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:43.657856 21634 hybrid_clock.cc:648] HybridClock initialized: now 1786372843657826 us; error 46 us; skew 500 ppm
Aug 10 14:40:43 dist-test-slave-rmhg krb5kdc[21102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372843, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:43.660543 21634 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:40:43.661492 21634 webserver.cc:460] Webserver started at http://127.15.84.3:36755/ using document root <none> and password file <none>
I20260810 14:40:43.662063 21634 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:43.662125 21634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:43.662335 21634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:43.664515 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "179f987b260a4fd1a5c8349577de9269"
format_stamp: "Formatted at 2026-08-10 14:40:43 on dist-test-slave-rmhg"
server_key: "f0de95ed2b0b2fb4631d8e2ef93ec523"
server_key_iv: "0df33fcb7e3d9c702c04d417aa58522a"
server_key_version: "encryptionkey@0"
I20260810 14:40:43.665096 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "179f987b260a4fd1a5c8349577de9269"
format_stamp: "Formatted at 2026-08-10 14:40:43 on dist-test-slave-rmhg"
server_key: "f0de95ed2b0b2fb4631d8e2ef93ec523"
server_key_iv: "0df33fcb7e3d9c702c04d417aa58522a"
server_key_version: "encryptionkey@0"
I20260810 14:40:43.669224 21634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:40:43.671977 21656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:43.673369 21634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:43.673488 21634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "179f987b260a4fd1a5c8349577de9269"
format_stamp: "Formatted at 2026-08-10 14:40:43 on dist-test-slave-rmhg"
server_key: "f0de95ed2b0b2fb4631d8e2ef93ec523"
server_key_iv: "0df33fcb7e3d9c702c04d417aa58522a"
server_key_version: "encryptionkey@0"
I20260810 14:40:43.673605 21634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:43.696834 21634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:43.699831 21634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:43.700027 21634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:43.700672 21634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:43.701707 21634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:43.701761 21634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:43.701802 21634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:43.701821 21634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:43.711238 21634 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:34499
I20260810 14:40:43.711266 21818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:34499 every 8 connection(s)
I20260810 14:40:43.712237 21634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:40:43.721449 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 21634
I20260810 14:40:43.721606 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:40:43.721949 15696 external_mini_cluster.cc:1428] Setting key daf4bfc70121059e4937a404d314ef09
Aug 10 14:40:43 dist-test-slave-rmhg krb5kdc[21102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372843, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:43.726558 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:43.714184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:44415 (local address 127.15.84.62:46779)
0810 14:40:43.714329 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:43.714332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:43.715280 (+ 948us) server_negotiation.cc:407] Connection header received
0810 14:40:43.716686 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:43.716689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:43.716750 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:43.716824 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:43.718094 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.718623 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.719797 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.720028 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.720124 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:43.723216 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:43.723219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:43.723221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:43.723248 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:43.724658 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:43.725187 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:43.725190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:43.725192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:43.725271 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:43.725593 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:43.725595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:43.725596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:43.725796 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:43.725870 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:43.726328 (+ 458us) server_negotiation.cc:299] Negotiation successful
0810 14:40:43.726435 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:40:43.727432 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:43.714599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:46779 (local address 127.15.84.3:44415)
0810 14:40:43.715121 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:43.715167 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:43.716456 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:43.717017 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:43.717024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:43.717443 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:43.717951 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:43.717965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.718748 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.718751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:43.719650 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:43.719662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.720198 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.720201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:43.720352 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:43.721202 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:43.721227 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:43.723038 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:43.724774 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:43.724780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:43.724793 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:43.725078 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:43.725366 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:43.725369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:43.725370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:43.725521 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:43.725869 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:43.725875 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:43.726137 (+ 262us) client_negotiation.cc:769] Sending connection context
0810 14:40:43.726297 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 14:40:43.726573 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":136,"threads_started":1}
I20260810 14:40:43.728551 21819 heartbeater.cc:347] Connected to a master server at 127.15.84.62:46779
I20260810 14:40:43.728801 21819 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:43.729319 21819 heartbeater.cc:511] Master 127.15.84.62:46779 requested a full tablet report, sending...
I20260810 14:40:43.730379 21163 ts_manager.cc:194] Registered new tserver with Master: 179f987b260a4fd1a5c8349577de9269 (127.15.84.3:34499)
I20260810 14:40:43.730928 21163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:44415
I20260810 14:40:43.735746 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 14:40:43 dist-test-slave-rmhg krb5kdc[21102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:40:43 dist-test-slave-rmhg krb5kdc[21102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.1@KRBTEST.COM
I20260810 14:40:43.762787 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:43.753769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.15.84.1:40501)
0810 14:40:43.754117 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:43.754145 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:43.754171 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:40:43.754225 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:43.754231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:43.754445 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:43.754638 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:43.755385 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.756183 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.756961 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.757405 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.757491 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:43.759790 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:43.759801 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:43.759817 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:43.759843 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:43.761259 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:43.761639 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:43.761646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:43.761648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:43.761729 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:43.761922 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:43.761924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:43.761925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:43.762189 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:43.762358 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:43.762502 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 14:40:43.762642 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":138,"threads_started":1}
Aug 10 14:40:43 dist-test-slave-rmhg krb5kdc[21102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:43.773769 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:43.765749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.15.84.62:46779)
0810 14:40:43.765885 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:43.765890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:43.765959 (+ 69us) server_negotiation.cc:407] Connection header received
0810 14:40:43.766085 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:43.766088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:43.766139 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:43.766212 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:43.766830 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.767358 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.768134 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:43.768369 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:43.768461 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:43.770892 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:43.770896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:43.770898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:43.770927 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:43.772325 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:43.772669 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:43.772671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:43.772673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:43.772746 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:43.772926 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:43.772928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:43.772929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:43.773195 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:43.773305 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:43.773530 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 14:40:43.773637 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 14:40:43.774194 21163 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43486
I20260810 14:40:43.774816 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 21252
I20260810 14:40:43.796334 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 21447
I20260810 14:40:43.818813 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 21634
I20260810 14:40:43.835070 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 21120
2026-08-10T14:40:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2372 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:43 dist-test-slave-rmhg krb5kdc[21835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:43 dist-test-slave-rmhg krb5kdc[21835](info): set up 2 sockets
Aug 10 14:40:43 dist-test-slave-rmhg krb5kdc[21835](info): commencing operation
krb5kdc: starting...
W20260810 14:40:44.977244 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:40:45 dist-test-slave-rmhg krb5kdc[21835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:40:45.112608 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:34149
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:34149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:45.228384 21858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:45.228713 21858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:45.228780 21858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:45.232916 21858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:45.233000 21858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:45.233027 21858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:45.233048 21858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:45.233068 21858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:45.238579 21858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:34149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:34149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:45.239732 21858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:45.240805 21858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:45.249171 21867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:45.249186 21863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:45.249182 21864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:45.249627 21858 server_base.cc:1034] running on GCE node
I20260810 14:40:45.250213 21858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:45.251341 21858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:45.252499 21858 hybrid_clock.cc:648] HybridClock initialized: now 1786372845252486 us; error 29 us; skew 500 ppm
Aug 10 14:40:45 dist-test-slave-rmhg krb5kdc[21835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372845, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:45.255282 21858 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:40:45.256037 21858 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:45.256107 21858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:45.256325 21858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:45.258493 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "a297b0d623894319ac4a6a468f06a1e3"
format_stamp: "Formatted at 2026-08-10 14:40:45 on dist-test-slave-rmhg"
server_key: "81e6eb4560131e87bec207edb4988fc8"
server_key_iv: "3ab1f8c35e819333c596b1e11c7aff0b"
server_key_version: "encryptionkey@0"
I20260810 14:40:45.259052 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "a297b0d623894319ac4a6a468f06a1e3"
format_stamp: "Formatted at 2026-08-10 14:40:45 on dist-test-slave-rmhg"
server_key: "81e6eb4560131e87bec207edb4988fc8"
server_key_iv: "3ab1f8c35e819333c596b1e11c7aff0b"
server_key_version: "encryptionkey@0"
I20260810 14:40:45.262763 21858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 14:40:45.265255 21878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:45.266515 21858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:40:45.266639 21858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "a297b0d623894319ac4a6a468f06a1e3"
format_stamp: "Formatted at 2026-08-10 14:40:45 on dist-test-slave-rmhg"
server_key: "81e6eb4560131e87bec207edb4988fc8"
server_key_iv: "3ab1f8c35e819333c596b1e11c7aff0b"
server_key_version: "encryptionkey@0"
I20260810 14:40:45.266747 21858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:45.280473 21858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:45.283851 21858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:45.284049 21858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:45.291182 21858 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:34149
I20260810 14:40:45.291417 21956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:34149 every 8 connection(s)
I20260810 14:40:45.291884 21858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:45.294852 21957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:45.300297 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 21858
I20260810 14:40:45.300446 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:45.300746 15696 external_mini_cluster.cc:1428] Setting key abccc16f4a3934ad94e82dc79eb2a5e2
I20260810 14:40:45.300806 21957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3: Bootstrap starting.
I20260810 14:40:45.303460 21957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:45.304387 21957 log.cc:824] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:45.306577 21957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3: No bootstrap required, opened a new log
Aug 10 14:40:45 dist-test-slave-rmhg krb5kdc[21835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:45.310992 21957 raft_consensus.cc:348] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a297b0d623894319ac4a6a468f06a1e3" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34149 } }
I20260810 14:40:45.311252 21957 raft_consensus.cc:374] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:45.311307 21957 raft_consensus.cc:729] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297b0d623894319ac4a6a468f06a1e3, State: Initialized, Role: FOLLOWER
I20260810 14:40:45.311792 21957 consensus_queue.cc:260] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a297b0d623894319ac4a6a468f06a1e3" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34149 } }
I20260810 14:40:45.311924 21957 raft_consensus.cc:388] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:45.311988 21957 raft_consensus.cc:482] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:45.312088 21957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:45.313241 21957 raft_consensus.cc:504] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a297b0d623894319ac4a6a468f06a1e3" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34149 } }
I20260810 14:40:45.313620 21957 leader_election.cc:302] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a297b0d623894319ac4a6a468f06a1e3; no voters:
I20260810 14:40:45.313971 21957 leader_election.cc:288] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:45.314153 21963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:45.314391 21963 raft_consensus.cc:686] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [term 1 LEADER]: Becoming Leader. State: Replica: a297b0d623894319ac4a6a468f06a1e3, State: Running, Role: LEADER
I20260810 14:40:45.314798 21963 consensus_queue.cc:237] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a297b0d623894319ac4a6a468f06a1e3" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34149 } }
I20260810 14:40:45.315490 21957 sys_catalog.cc:564] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:45.316232 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:45.302108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.15.84.62:34149)
0810 14:40:45.302518 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:45.302535 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:45.302573 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:40:45.303228 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:45.303247 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:45.303609 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:45.303908 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:45.304826 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:45.305976 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:45.307125 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:45.307608 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:45.307728 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:45.310572 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:45.310578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:45.310593 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:45.310625 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:45.312403 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:45.312886 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:45.312891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:45.312895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:45.313007 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:45.313341 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:45.313346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:45.313348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:45.313887 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:45.314108 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:45.314746 (+ 638us) server_negotiation.cc:299] Negotiation successful
0810 14:40:45.315105 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20260810 14:40:45.317535 21964 sys_catalog.cc:455] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a297b0d623894319ac4a6a468f06a1e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a297b0d623894319ac4a6a468f06a1e3" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34149 } } }
I20260810 14:40:45.317544 21965 sys_catalog.cc:455] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a297b0d623894319ac4a6a468f06a1e3. Latest consensus state: current_term: 1 leader_uuid: "a297b0d623894319ac4a6a468f06a1e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a297b0d623894319ac4a6a468f06a1e3" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34149 } } }
I20260810 14:40:45.317782 21965 sys_catalog.cc:458] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:45.317823 21964 sys_catalog.cc:458] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3 [sys.catalog]: This master's current role is: LEADER
W20260810 14:40:45.320832 21983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:40:45.320905 21983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:40:45.320987 21986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:45.321834 21986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:45.328200 21986 catalog_manager.cc:1347] Generated new cluster ID: c85e4f3dbceb414ea42ad0470ce98476
I20260810 14:40:45.328293 21986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:45.343799 21986 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:45.345312 21986 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:45.355103 21986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3: Generated new TSK 0
I20260810 14:40:45.355907 21986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:40:45.403967 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34149
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:45.525007 21995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:45.525408 21995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:45.525483 21995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:45.529821 21995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:45.529912 21995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:45.530005 21995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:40:45.535719 21995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:45.536917 21995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:45.538019 21995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:45.548022 22001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:45.548025 22012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:45.548024 22004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:45.548533 21995 server_base.cc:1034] running on GCE node
I20260810 14:40:45.549099 21995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:45.550053 21995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:45.551273 21995 hybrid_clock.cc:648] HybridClock initialized: now 1786372845551244 us; error 49 us; skew 500 ppm
Aug 10 14:40:45 dist-test-slave-rmhg krb5kdc[21835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372845, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:45.559501 21995 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:40:45.560384 21995 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:45.560461 21995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:45.560688 21995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:45.563060 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "0bea3a46d8df4eda868c50ee7b67de18"
format_stamp: "Formatted at 2026-08-10 14:40:45 on dist-test-slave-rmhg"
server_key: "6bcf64b33ca6561a8ddb9efb1cd49085"
server_key_iv: "9e4e9dbd7b93f592042c084b776d2395"
server_key_version: "encryptionkey@0"
I20260810 14:40:45.563633 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "0bea3a46d8df4eda868c50ee7b67de18"
format_stamp: "Formatted at 2026-08-10 14:40:45 on dist-test-slave-rmhg"
server_key: "6bcf64b33ca6561a8ddb9efb1cd49085"
server_key_iv: "9e4e9dbd7b93f592042c084b776d2395"
server_key_version: "encryptionkey@0"
I20260810 14:40:45.567602 21995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 14:40:45.570297 22020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:45.571972 21995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:45.572108 21995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "0bea3a46d8df4eda868c50ee7b67de18"
format_stamp: "Formatted at 2026-08-10 14:40:45 on dist-test-slave-rmhg"
server_key: "6bcf64b33ca6561a8ddb9efb1cd49085"
server_key_iv: "9e4e9dbd7b93f592042c084b776d2395"
server_key_version: "encryptionkey@0"
I20260810 14:40:45.572239 21995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:45.583127 21995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:45.586651 21995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:45.586850 21995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:45.587509 21995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:45.588451 21995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:45.588501 21995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:45.588546 21995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:45.588577 21995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:45.597985 21995 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:42339
I20260810 14:40:45.598006 22195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:42339 every 8 connection(s)
I20260810 14:40:45.598685 21995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:40:45.602468 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 21995
I20260810 14:40:45.602594 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:40:45.602864 15696 external_mini_cluster.cc:1428] Setting key 41e54e99168c7c30a7f1b4d136febaaf
Aug 10 14:40:45 dist-test-slave-rmhg krb5kdc[21835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372845, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:45.613027 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:45.600517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:41865 (local address 127.15.84.62:34149)
0810 14:40:45.600663 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:45.600667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:45.601562 (+ 895us) server_negotiation.cc:407] Connection header received
0810 14:40:45.602947 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:45.602951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:45.603013 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:45.603088 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:45.604540 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:45.605100 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:45.606299 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:45.606531 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:45.606622 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:45.609216 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:45.609219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:45.609222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:45.609253 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:45.610844 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:45.611365 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:45.611369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:45.611371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:45.611469 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:45.611771 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:45.611775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:45.611776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:45.612020 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:45.612123 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:45.612712 (+ 589us) server_negotiation.cc:299] Negotiation successful
0810 14:40:45.612851 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:40:45.613879 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:45.600883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:34149 (local address 127.15.84.1:41865)
0810 14:40:45.601405 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:45.601451 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:45.602719 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:45.603295 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:45.603303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:45.603773 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:45.604385 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:45.604402 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:45.605249 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:45.605252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:45.606174 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:45.606184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:45.606681 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:45.606685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:45.606858 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:45.607458 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:45.607475 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:45.609004 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:45.610967 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:45.610973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:45.610996 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:45.611269 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:45.611554 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:45.611556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:45.611557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:45.611689 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:45.612157 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:45.612163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:45.612444 (+ 281us) client_negotiation.cc:769] Sending connection context
0810 14:40:45.612629 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 14:40:45.612922 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:40:45.615274 22196 heartbeater.cc:347] Connected to a master server at 127.15.84.62:34149
I20260810 14:40:45.615567 22196 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:45.616127 22196 heartbeater.cc:511] Master 127.15.84.62:34149 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
I20260810 14:40:45.618162 21905 ts_manager.cc:194] Registered new tserver with Master: 0bea3a46d8df4eda868c50ee7b67de18 (127.15.84.1:42339)
I20260810 14:40:45.619545 21905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:41865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:40:45.634388 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34149
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:45.754671 22202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:45.754961 22202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:45.755024 22202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:45.759255 22202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:45.759337 22202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:45.759424 22202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:40:45.765048 22202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:45.766219 22202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:45.767275 22202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:45.777484 22212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:45.777482 22210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:45.777475 22209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:45.777763 22202 server_base.cc:1034] running on GCE node
I20260810 14:40:45.778179 22202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:45.778857 22202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:45.780035 22202 hybrid_clock.cc:648] HybridClock initialized: now 1786372845780008 us; error 41 us; skew 500 ppm
Aug 10 14:40:45 dist-test-slave-rmhg krb5kdc[21835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372845, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:45.783082 22202 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:40:45.783885 22202 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:45.783959 22202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:45.784163 22202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:45.786375 22202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "c58e17ddb3e34315b2ff62ff2922c1ab"
format_stamp: "Formatted at 2026-08-10 14:40:45 on dist-test-slave-rmhg"
server_key: "e46ea29dc81db07c3f004c3993dd0912"
server_key_iv: "7db43ad4198999b330169b7cee152e8a"
server_key_version: "encryptionkey@0"
I20260810 14:40:45.787046 22202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "c58e17ddb3e34315b2ff62ff2922c1ab"
format_stamp: "Formatted at 2026-08-10 14:40:45 on dist-test-slave-rmhg"
server_key: "e46ea29dc81db07c3f004c3993dd0912"
server_key_iv: "7db43ad4198999b330169b7cee152e8a"
server_key_version: "encryptionkey@0"
I20260810 14:40:45.791013 22202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:40:45.793596 22221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:45.794884 22202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:40:45.794999 22202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "c58e17ddb3e34315b2ff62ff2922c1ab"
format_stamp: "Formatted at 2026-08-10 14:40:45 on dist-test-slave-rmhg"
server_key: "e46ea29dc81db07c3f004c3993dd0912"
server_key_iv: "7db43ad4198999b330169b7cee152e8a"
server_key_version: "encryptionkey@0"
I20260810 14:40:45.795104 22202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:45.817308 22202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:45.820782 22202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:45.820942 22202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:45.821548 22202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:45.822443 22202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:45.822494 22202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:45.822535 22202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:45.822556 22202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:45.831586 22202 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:46061
I20260810 14:40:45.831619 22381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:46061 every 8 connection(s)
I20260810 14:40:45.832283 22202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
Aug 10 14:40:45 dist-test-slave-rmhg krb5kdc[21835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372845, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:45.842161 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 22202
I20260810 14:40:45.842269 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:40:45.842537 15696 external_mini_cluster.cc:1428] Setting key ce4488b7e2379a56152a6613b9f72338
I20260810 14:40:45.846208 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:45.834241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:50367 (local address 127.15.84.62:34149)
0810 14:40:45.834393 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:45.834398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:45.835354 (+ 956us) server_negotiation.cc:407] Connection header received
0810 14:40:45.836620 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:45.836623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:45.836673 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:45.836757 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:45.838073 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:45.838622 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:45.839773 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:45.840008 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:45.840096 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:45.842711 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:45.842714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:45.842717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:45.842742 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:45.844075 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:45.844604 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:45.844607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:45.844608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:45.844694 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:45.845028 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:45.845032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:45.845033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:45.845267 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:45.845355 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:45.845908 (+ 553us) server_negotiation.cc:299] Negotiation successful
0810 14:40:45.846041 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:40:45.847153 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:45.834686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:34149 (local address 127.15.84.2:50367)
0810 14:40:45.835196 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:45.835243 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:45.836420 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:45.836918 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:45.836924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:45.837371 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:45.837920 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:45.837935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:45.838730 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:45.838732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:45.839632 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:45.839641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:45.840141 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:45.840144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:45.840300 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:45.840889 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:45.840906 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:45.842536 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:45.844197 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:45.844204 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:45.844221 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:45.844507 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:45.844781 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:45.844784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:45.844786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:45.844938 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:45.845378 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:45.845386 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:45.845664 (+ 278us) client_negotiation.cc:769] Sending connection context
0810 14:40:45.845846 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 14:40:45.846227 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":135,"threads_started":1}
I20260810 14:40:45.848315 22382 heartbeater.cc:347] Connected to a master server at 127.15.84.62:34149
I20260810 14:40:45.848565 22382 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:45.849043 22382 heartbeater.cc:511] Master 127.15.84.62:34149 requested a full tablet report, sending...
I20260810 14:40:45.850210 21905 ts_manager.cc:194] Registered new tserver with Master: c58e17ddb3e34315b2ff62ff2922c1ab (127.15.84.2:46061)
I20260810 14:40:45.851001 21905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:50367
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:40:45.872506 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34149
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:45.989315 22389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:45.989595 22389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:45.989643 22389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:45.993714 22389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:45.993783 22389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:45.993845 22389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:40:45.999368 22389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:46.000450 22389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:46.001495 22389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:46.011214 22396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:46.011225 22395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:46.011221 22400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:46.011287 22389 server_base.cc:1034] running on GCE node
I20260810 14:40:46.011888 22389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:46.012538 22389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:46.013707 22389 hybrid_clock.cc:648] HybridClock initialized: now 1786372846013689 us; error 31 us; skew 500 ppm
Aug 10 14:40:46 dist-test-slave-rmhg krb5kdc[21835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372846, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:46.016420 22389 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:40:46.017230 22389 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:46.017305 22389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:46.017517 22389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:46.019578 22389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "504b2529b5ae4341a7a618ffa1fa2dfa"
format_stamp: "Formatted at 2026-08-10 14:40:46 on dist-test-slave-rmhg"
server_key: "07854aa5585faa1fe403b679da907aef"
server_key_iv: "95f29708f7773d358626f14dcf555575"
server_key_version: "encryptionkey@0"
I20260810 14:40:46.020123 22389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "504b2529b5ae4341a7a618ffa1fa2dfa"
format_stamp: "Formatted at 2026-08-10 14:40:46 on dist-test-slave-rmhg"
server_key: "07854aa5585faa1fe403b679da907aef"
server_key_iv: "95f29708f7773d358626f14dcf555575"
server_key_version: "encryptionkey@0"
I20260810 14:40:46.023738 22389 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 14:40:46.026183 22407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:46.027419 22389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:40:46.027532 22389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "504b2529b5ae4341a7a618ffa1fa2dfa"
format_stamp: "Formatted at 2026-08-10 14:40:46 on dist-test-slave-rmhg"
server_key: "07854aa5585faa1fe403b679da907aef"
server_key_iv: "95f29708f7773d358626f14dcf555575"
server_key_version: "encryptionkey@0"
I20260810 14:40:46.027628 22389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:46.052294 22389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:46.055444 22389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:46.055641 22389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:46.056270 22389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:46.057188 22389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:46.057241 22389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:46.057298 22389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:46.057322 22389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:46.066207 22389 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:46211
I20260810 14:40:46.066236 22574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:46211 every 8 connection(s)
I20260810 14:40:46.066805 22389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:40:46.070360 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 22389
I20260810 14:40:46.070513 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:40:46.070785 15696 external_mini_cluster.cc:1428] Setting key 2daf608f72758035ce299c53f0ba50c5
Aug 10 14:40:46 dist-test-slave-rmhg krb5kdc[21835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372846, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:46.080500 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.068733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:57283 (local address 127.15.84.62:34149)
0810 14:40:46.068870 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:46.068875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:46.069801 (+ 926us) server_negotiation.cc:407] Connection header received
0810 14:40:46.071276 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:46.071280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:46.071345 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:46.071430 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:46.072817 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.073459 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.074649 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.074881 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.074968 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:46.077509 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:46.077512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:46.077514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:46.077540 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:46.078756 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:46.079209 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:46.079211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:46.079212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:46.079281 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:46.079546 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:46.079548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:46.079549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:46.079731 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:46.079803 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:46.080273 (+ 470us) server_negotiation.cc:299] Negotiation successful
0810 14:40:46.080375 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:40:46.081399 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.069094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:34149 (local address 127.15.84.3:57283)
0810 14:40:46.069654 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:46.069711 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:46.071050 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:46.071633 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:46.071642 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:46.072130 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:46.072658 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:46.072674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.073601 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.073605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:46.074528 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:46.074539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.075033 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.075037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:46.075254 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:46.075763 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:46.075780 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:46.077350 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:46.078859 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:46.078864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:46.078876 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:46.079124 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:46.079345 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:46.079347 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:46.079348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:46.079479 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:46.079804 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:46.079809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:46.080065 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 14:40:46.080236 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 14:40:46.080499 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":122,"threads_started":1}
I20260810 14:40:46.082540 22576 heartbeater.cc:347] Connected to a master server at 127.15.84.62:34149
I20260810 14:40:46.082790 22576 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:46.083262 22576 heartbeater.cc:511] Master 127.15.84.62:34149 requested a full tablet report, sending...
I20260810 14:40:46.084379 21905 ts_manager.cc:194] Registered new tserver with Master: 504b2529b5ae4341a7a618ffa1fa2dfa (127.15.84.3:46211)
I20260810 14:40:46.085002 21905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:57283
I20260810 14:40:46.096845 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:40:46.106127 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.098855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.15.84.62:34149)
0810 14:40:46.098985 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:46.098990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:46.099148 (+ 158us) server_negotiation.cc:407] Connection header received
0810 14:40:46.099315 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:46.099319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:46.099376 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:46.099446 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:46.100183 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.100749 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.101596 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.101823 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.101905 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:46.103133 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:46.103136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:46.103138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:46.103167 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:46.104635 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:46.104987 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:46.104990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:46.104991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:46.105078 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:46.105292 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:46.105294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:46.105295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:46.105559 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:46.105664 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:46.105849 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 14:40:46.105958 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 14:40:46.109069 21905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:40:46.112200 21905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:46.126876 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.120712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:46061 (local address 127.0.0.1:40274)
0810 14:40:46.121718 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:46.121731 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:46.121828 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:46.122574 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:46.122581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:46.122617 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:46.122839 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:46.122846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.124331 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.124333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:46.125491 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:46.125501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.126310 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.126314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:46.126435 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:46.126477 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 14:40:46.126583 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 14:40:46.126714 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":123,"spinlock_wait_cycles":6272,"thread_start_us":85,"threads_started":1}
I20260810 14:40:46.126880 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.120893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.15.84.2:46061)
0810 14:40:46.121244 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:46.121259 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:46.121934 (+ 675us) server_negotiation.cc:407] Connection header received
0810 14:40:46.122034 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:46.122039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:46.122240 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:46.122386 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:46.122967 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.124190 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.125611 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.126183 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.126286 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:46.126454 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:46.126627 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 14:40:46.126728 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":117,"threads_started":1}
I20260810 14:40:46.127303 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.121412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:46211 (local address 127.0.0.1:51874)
0810 14:40:46.122349 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:46.122360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:46.122445 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:46.122888 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:46.122891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:46.122908 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:46.123003 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:46.123009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.123976 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.123980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:46.125497 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:46.125509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.126818 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.126822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:46.126985 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:46.127024 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 14:40:46.127093 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 14:40:46.127164 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":93,"thread_start_us":91,"threads_started":1}
I20260810 14:40:46.127776 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.121607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.15.84.3:46211)
0810 14:40:46.122035 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:46.122051 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:46.122539 (+ 488us) server_negotiation.cc:407] Connection header received
0810 14:40:46.122607 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:46.122610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:46.122765 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:46.122894 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:46.123107 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.123851 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.125681 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.126508 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.126614 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:46.126752 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:46.127527 (+ 775us) server_negotiation.cc:299] Negotiation successful
0810 14:40:46.127630 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":137,"threads_started":1}
I20260810 14:40:46.130024 22292 tablet_service.cc:1467] Processing CreateTablet for tablet 44bcbd9b6a9b46609e107cdf5551518c (DEFAULT_TABLE table=test-table [id=1c163b2a48f0440a8c85da7ffe3f2822]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:46.130213 22475 tablet_service.cc:1467] Processing CreateTablet for tablet 44bcbd9b6a9b46609e107cdf5551518c (DEFAULT_TABLE table=test-table [id=1c163b2a48f0440a8c85da7ffe3f2822]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:46.131047 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bcbd9b6a9b46609e107cdf5551518c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:46.131147 22475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bcbd9b6a9b46609e107cdf5551518c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:46.134636 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.120645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:42339 (local address 127.0.0.1:57948)
0810 14:40:46.121382 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:46.121437 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:46.121567 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:46.122574 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:46.122581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:46.122618 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:46.122825 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:46.122833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.124311 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.124314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:46.125533 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:46.125540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.126818 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.126822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:46.126981 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:46.127027 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 14:40:46.127092 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 14:40:46.134469 (+ 7377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":7309,"spinlock_wait_cycles":8576,"thread_start_us":84,"threads_started":1}
I20260810 14:40:46.135423 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.121495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57948 (local address 127.15.84.1:42339)
0810 14:40:46.121934 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:46.121940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:46.121958 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:40:46.122026 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:46.122031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:46.122240 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:46.122387 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:46.122967 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.124181 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.125662 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.126508 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.126614 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:46.126752 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:46.135056 (+ 8304us) server_negotiation.cc:299] Negotiation successful
0810 14:40:46.135258 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":124,"threads_started":1}
I20260810 14:40:46.138000 22605 tablet_bootstrap.cc:492] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab: Bootstrap starting.
I20260810 14:40:46.138818 22094 tablet_service.cc:1467] Processing CreateTablet for tablet 44bcbd9b6a9b46609e107cdf5551518c (DEFAULT_TABLE table=test-table [id=1c163b2a48f0440a8c85da7ffe3f2822]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:46.139496 22606 tablet_bootstrap.cc:492] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa: Bootstrap starting.
I20260810 14:40:46.140015 22094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bcbd9b6a9b46609e107cdf5551518c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:46.140025 22605 tablet_bootstrap.cc:654] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:46.141067 22605 log.cc:824] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:46.141762 22606 tablet_bootstrap.cc:654] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:46.143003 22606 log.cc:824] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:46.143365 22605 tablet_bootstrap.cc:492] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab: No bootstrap required, opened a new log
I20260810 14:40:46.143579 22605 ts_tablet_manager.cc:1397] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260810 14:40:46.145318 22606 tablet_bootstrap.cc:492] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa: No bootstrap required, opened a new log
I20260810 14:40:46.145525 22606 ts_tablet_manager.cc:1397] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:40:46.146786 22605 raft_consensus.cc:348] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bea3a46d8df4eda868c50ee7b67de18" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42339 } } peers { permanent_uuid: "c58e17ddb3e34315b2ff62ff2922c1ab" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46061 } } peers { permanent_uuid: "504b2529b5ae4341a7a618ffa1fa2dfa" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46211 } }
I20260810 14:40:46.147037 22605 raft_consensus.cc:374] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:46.147099 22605 raft_consensus.cc:729] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c58e17ddb3e34315b2ff62ff2922c1ab, State: Initialized, Role: FOLLOWER
I20260810 14:40:46.147621 22605 consensus_queue.cc:260] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bea3a46d8df4eda868c50ee7b67de18" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42339 } } peers { permanent_uuid: "c58e17ddb3e34315b2ff62ff2922c1ab" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46061 } } peers { permanent_uuid: "504b2529b5ae4341a7a618ffa1fa2dfa" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46211 } }
I20260810 14:40:46.148593 22612 tablet_bootstrap.cc:492] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18: Bootstrap starting.
I20260810 14:40:46.148705 22605 ts_tablet_manager.cc:1428] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 14:40:46.149091 22382 heartbeater.cc:503] Master 127.15.84.62:34149 was elected leader, sending a full tablet report...
I20260810 14:40:46.149380 22606 raft_consensus.cc:348] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bea3a46d8df4eda868c50ee7b67de18" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42339 } } peers { permanent_uuid: "c58e17ddb3e34315b2ff62ff2922c1ab" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46061 } } peers { permanent_uuid: "504b2529b5ae4341a7a618ffa1fa2dfa" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46211 } }
I20260810 14:40:46.149696 22606 raft_consensus.cc:374] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:46.149765 22606 raft_consensus.cc:729] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504b2529b5ae4341a7a618ffa1fa2dfa, State: Initialized, Role: FOLLOWER
I20260810 14:40:46.150279 22606 consensus_queue.cc:260] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bea3a46d8df4eda868c50ee7b67de18" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42339 } } peers { permanent_uuid: "c58e17ddb3e34315b2ff62ff2922c1ab" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46061 } } peers { permanent_uuid: "504b2529b5ae4341a7a618ffa1fa2dfa" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46211 } }
I20260810 14:40:46.150986 22576 heartbeater.cc:503] Master 127.15.84.62:34149 was elected leader, sending a full tablet report...
I20260810 14:40:46.151131 22612 tablet_bootstrap.cc:654] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:46.151367 22606 ts_tablet_manager.cc:1428] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:40:46.152033 22612 log.cc:824] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:46.154656 22612 tablet_bootstrap.cc:492] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18: No bootstrap required, opened a new log
I20260810 14:40:46.154886 22612 ts_tablet_manager.cc:1397] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:40:46.158463 22612 raft_consensus.cc:348] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bea3a46d8df4eda868c50ee7b67de18" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42339 } } peers { permanent_uuid: "c58e17ddb3e34315b2ff62ff2922c1ab" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46061 } } peers { permanent_uuid: "504b2529b5ae4341a7a618ffa1fa2dfa" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46211 } }
I20260810 14:40:46.158715 22612 raft_consensus.cc:374] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:46.158751 22612 raft_consensus.cc:729] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bea3a46d8df4eda868c50ee7b67de18, State: Initialized, Role: FOLLOWER
I20260810 14:40:46.159205 22612 consensus_queue.cc:260] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bea3a46d8df4eda868c50ee7b67de18" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42339 } } peers { permanent_uuid: "c58e17ddb3e34315b2ff62ff2922c1ab" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46061 } } peers { permanent_uuid: "504b2529b5ae4341a7a618ffa1fa2dfa" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46211 } }
I20260810 14:40:46.159927 22196 heartbeater.cc:503] Master 127.15.84.62:34149 was elected leader, sending a full tablet report...
I20260810 14:40:46.160151 22612 ts_tablet_manager.cc:1428] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 14:40:46.237771 22616 raft_consensus.cc:482] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:46.238013 22616 raft_consensus.cc:504] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bea3a46d8df4eda868c50ee7b67de18" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42339 } } peers { permanent_uuid: "c58e17ddb3e34315b2ff62ff2922c1ab" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46061 } } peers { permanent_uuid: "504b2529b5ae4341a7a618ffa1fa2dfa" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46211 } }
I20260810 14:40:46.239161 22616 leader_election.cc:288] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c58e17ddb3e34315b2ff62ff2922c1ab (127.15.84.2:46061), 504b2529b5ae4341a7a618ffa1fa2dfa (127.15.84.3:46211)
I20260810 14:40:46.243253 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.239587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:51145 (local address 127.15.84.3:46211)
0810 14:40:46.239734 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:46.239739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:46.240050 (+ 311us) server_negotiation.cc:407] Connection header received
0810 14:40:46.240168 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:46.240179 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:46.240234 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:46.240311 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:46.240592 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.241090 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.242284 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.242677 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.242763 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:46.242835 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:46.243038 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 14:40:46.243110 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 14:40:46.243263 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.239587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:32987 (local address 127.15.84.2:46061)
0810 14:40:46.239734 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:46.239739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:46.239996 (+ 257us) server_negotiation.cc:407] Connection header received
0810 14:40:46.240100 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:46.240104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:46.240162 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:46.240224 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:46.240583 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.241091 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.242281 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.242690 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.242785 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:46.242850 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:46.243069 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 14:40:46.243142 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:40:46.243283 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.239466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:46061 (local address 127.15.84.1:32987)
0810 14:40:46.239882 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:46.239894 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:46.240003 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:46.240258 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:46.240261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:46.240277 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:46.240468 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:46.240475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.241210 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.241214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:46.242157 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:46.242167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.242804 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.242807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:46.242947 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:46.242966 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 14:40:46.243037 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 14:40:46.243109 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":115,"thread_start_us":138,"threads_started":1}
I20260810 14:40:46.243292 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.239466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:46211 (local address 127.15.84.1:51145)
0810 14:40:46.239925 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:46.239937 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:46.240067 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:46.240331 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:46.240334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:46.240350 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:46.240480 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:46.240484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.241212 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.241216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:46.242159 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:46.242169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.242794 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.242798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:46.242919 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:46.242940 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 14:40:46.242999 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 14:40:46.243069 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":88,"thread_start_us":104,"threads_started":1}
I20260810 14:40:46.244125 22504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bcbd9b6a9b46609e107cdf5551518c" candidate_uuid: "0bea3a46d8df4eda868c50ee7b67de18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504b2529b5ae4341a7a618ffa1fa2dfa" is_pre_election: true
I20260810 14:40:46.244130 22316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bcbd9b6a9b46609e107cdf5551518c" candidate_uuid: "0bea3a46d8df4eda868c50ee7b67de18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c58e17ddb3e34315b2ff62ff2922c1ab" is_pre_election: true
I20260810 14:40:46.244424 22316 raft_consensus.cc:2454] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bea3a46d8df4eda868c50ee7b67de18 in term 0.
I20260810 14:40:46.244431 22504 raft_consensus.cc:2454] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bea3a46d8df4eda868c50ee7b67de18 in term 0.
I20260810 14:40:46.244925 22021 leader_election.cc:302] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bea3a46d8df4eda868c50ee7b67de18, 504b2529b5ae4341a7a618ffa1fa2dfa; no voters:
I20260810 14:40:46.245227 22616 raft_consensus.cc:2781] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:40:46.245325 22616 raft_consensus.cc:482] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:40:46.245376 22616 raft_consensus.cc:3037] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:46.246747 22616 raft_consensus.cc:504] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bea3a46d8df4eda868c50ee7b67de18" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42339 } } peers { permanent_uuid: "c58e17ddb3e34315b2ff62ff2922c1ab" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46061 } } peers { permanent_uuid: "504b2529b5ae4341a7a618ffa1fa2dfa" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46211 } }
I20260810 14:40:46.247084 22616 leader_election.cc:288] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [CANDIDATE]: Term 1 election: Requested vote from peers c58e17ddb3e34315b2ff62ff2922c1ab (127.15.84.2:46061), 504b2529b5ae4341a7a618ffa1fa2dfa (127.15.84.3:46211)
I20260810 14:40:46.247502 22316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bcbd9b6a9b46609e107cdf5551518c" candidate_uuid: "0bea3a46d8df4eda868c50ee7b67de18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c58e17ddb3e34315b2ff62ff2922c1ab"
I20260810 14:40:46.247628 22316 raft_consensus.cc:3037] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:46.248761 22316 raft_consensus.cc:2454] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bea3a46d8df4eda868c50ee7b67de18 in term 1.
I20260810 14:40:46.249053 22504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bcbd9b6a9b46609e107cdf5551518c" candidate_uuid: "0bea3a46d8df4eda868c50ee7b67de18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504b2529b5ae4341a7a618ffa1fa2dfa"
I20260810 14:40:46.249109 22023 leader_election.cc:302] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bea3a46d8df4eda868c50ee7b67de18, c58e17ddb3e34315b2ff62ff2922c1ab; no voters:
I20260810 14:40:46.249218 22504 raft_consensus.cc:3037] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:46.249336 22616 raft_consensus.cc:2781] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:46.249603 22616 raft_consensus.cc:686] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [term 1 LEADER]: Becoming Leader. State: Replica: 0bea3a46d8df4eda868c50ee7b67de18, State: Running, Role: LEADER
I20260810 14:40:46.250020 22616 consensus_queue.cc:237] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bea3a46d8df4eda868c50ee7b67de18" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42339 } } peers { permanent_uuid: "c58e17ddb3e34315b2ff62ff2922c1ab" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46061 } } peers { permanent_uuid: "504b2529b5ae4341a7a618ffa1fa2dfa" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46211 } }
I20260810 14:40:46.250643 22504 raft_consensus.cc:2454] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bea3a46d8df4eda868c50ee7b67de18 in term 1.
I20260810 14:40:46.253602 21905 catalog_manager.cc:5495] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bea3a46d8df4eda868c50ee7b67de18 (127.15.84.1). New cstate: current_term: 1 leader_uuid: "0bea3a46d8df4eda868c50ee7b67de18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bea3a46d8df4eda868c50ee7b67de18" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c58e17ddb3e34315b2ff62ff2922c1ab" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "504b2529b5ae4341a7a618ffa1fa2dfa" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20260810 14:40:46.284255 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.281039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.15.84.1:42339)
0810 14:40:46.281214 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:46.281218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:46.281235 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:40:46.281351 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:46.281353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:46.281411 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:46.281506 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:46.281712 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.282210 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.282987 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:46.283189 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:46.283257 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:46.283510 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:46.283955 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:46.284045 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:46.284089 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 14:40:46.284140 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:40:46.290606 22316 raft_consensus.cc:1261] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab [term 1 FOLLOWER]: Refusing update from remote peer 0bea3a46d8df4eda868c50ee7b67de18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:46.290606 22504 raft_consensus.cc:1261] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa [term 1 FOLLOWER]: Refusing update from remote peer 0bea3a46d8df4eda868c50ee7b67de18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:46.291420 22619 consensus_queue.cc:1035] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504b2529b5ae4341a7a618ffa1fa2dfa" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:46.291611 22616 consensus_queue.cc:1035] T 44bcbd9b6a9b46609e107cdf5551518c P 0bea3a46d8df4eda868c50ee7b67de18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c58e17ddb3e34315b2ff62ff2922c1ab" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:46.300086 22622 mvcc.cc:205] Tried to move back new op lower bound from 7316983178398486528 to 7316983178243969024. Current Snapshot: MvccSnapshot[applied={T|T < 7316983178398486528}]
I20260810 14:40:46.300477 22623 mvcc.cc:205] Tried to move back new op lower bound from 7316983178398486528 to 7316983178243969024. Current Snapshot: MvccSnapshot[applied={T|T < 7316983178398486528}]
I20260810 14:40:46.312883 21905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:46.313076 21905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:46.315857 21905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a297b0d623894319ac4a6a468f06a1e3: Sending DeleteTablet for 3 replicas of tablet 44bcbd9b6a9b46609e107cdf5551518c
I20260810 14:40:46.316787 22292 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bcbd9b6a9b46609e107cdf5551518c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:46 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:40274
I20260810 14:40:46.316859 22094 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bcbd9b6a9b46609e107cdf5551518c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:46 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:57948
I20260810 14:40:46.316833 22475 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bcbd9b6a9b46609e107cdf5551518c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:46 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:51874
I20260810 14:40:46.317955 22637 tablet_replica.cc:330] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab: stopping tablet replica
I20260810 14:40:46.318357 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 21995
I20260810 14:40:46.318356 22637 raft_consensus.cc:2229] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:46.318694 22637 raft_consensus.cc:2258] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:46.319072 22638 tablet_replica.cc:330] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa: stopping tablet replica
I20260810 14:40:46.319340 22638 raft_consensus.cc:2229] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:46.319594 22638 raft_consensus.cc:2258] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:46.322541 22638 ts_tablet_manager.cc:1905] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:46.322557 22637 ts_tablet_manager.cc:1905] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:46.324892 22638 ts_tablet_manager.cc:1918] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:46.324999 22638 log.cc:1196] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/44bcbd9b6a9b46609e107cdf5551518c
I20260810 14:40:46.325359 22638 ts_tablet_manager.cc:1939] T 44bcbd9b6a9b46609e107cdf5551518c P 504b2529b5ae4341a7a618ffa1fa2dfa: Deleting consensus metadata
I20260810 14:40:46.325654 22637 ts_tablet_manager.cc:1918] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:46.325788 22637 log.cc:1196] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/44bcbd9b6a9b46609e107cdf5551518c
I20260810 14:40:46.326071 22637 ts_tablet_manager.cc:1939] T 44bcbd9b6a9b46609e107cdf5551518c P c58e17ddb3e34315b2ff62ff2922c1ab: Deleting consensus metadata
I20260810 14:40:46.326869 21879 catalog_manager.cc:4841] TS 504b2529b5ae4341a7a618ffa1fa2dfa (127.15.84.3:46211): tablet 44bcbd9b6a9b46609e107cdf5551518c (table test-table [id=1c163b2a48f0440a8c85da7ffe3f2822]) successfully deleted
I20260810 14:40:46.327244 21886 catalog_manager.cc:4841] TS c58e17ddb3e34315b2ff62ff2922c1ab (127.15.84.2:46061): tablet 44bcbd9b6a9b46609e107cdf5551518c (table test-table [id=1c163b2a48f0440a8c85da7ffe3f2822]) successfully deleted
W20260810 14:40:46.333247 21879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:42339 (error 108)
I20260810 14:40:46.334091 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 22202
I20260810 14:40:46.334367 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.334055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:42339 (local address 127.0.0.1:57956)
0810 14:40:46.334204 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:46.334277 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:42339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 14:40:46.349000 21879 catalog_manager.cc:4584] TS 0bea3a46d8df4eda868c50ee7b67de18 (127.15.84.1:42339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44bcbd9b6a9b46609e107cdf5551518c: Network error: Client connection negotiation failed: client connection to 127.15.84.1:42339: connect: Connection refused (error 111)
I20260810 14:40:46.355247 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 22389
I20260810 14:40:46.374701 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 21858
2026-08-10T14:40:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2548 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T14:40:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:46Z Disabled control of system clock
I20260810 14:40:46.422861 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:46107
--webserver_interface=127.15.84.62
--webserver_port=0
--builtin_ntp_servers=127.15.84.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:46107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 14:40:46.540683 22648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:46.540994 22648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:46.541033 22648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:46.545212 22648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:46.545296 22648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:46.545310 22648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:46.545326 22648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:46.545341 22648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:46.550349 22648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:46107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:46107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:46.551839 22648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:46.553766 22648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:46.562620 22659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:46.562620 22655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:46.562664 22657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:46.562986 22648 server_base.cc:1034] running on GCE node
I20260810 14:40:46.563611 22648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:46.564850 22648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:46.566007 22648 hybrid_clock.cc:648] HybridClock initialized: now 1786372846565963 us; error 61 us; skew 500 ppm
I20260810 14:40:46.568054 22648 webserver.cc:460] Webserver started at http://127.15.84.62:41235/ using document root <none> and password file <none>
I20260810 14:40:46.568688 22648 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:46.568765 22648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:46.568989 22648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:46.571254 22648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "6e7fe2aa2a1a432e9bb72e3715a538fa"
format_stamp: "Formatted at 2026-08-10 14:40:46 on dist-test-slave-rmhg"
server_key: "fa7fc2e34dcde34f9988589276e35cec"
server_key_iv: "3bf5e25ad609518586ef3f2190162a31"
server_key_version: "encryptionkey@0"
I20260810 14:40:46.571830 22648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "6e7fe2aa2a1a432e9bb72e3715a538fa"
format_stamp: "Formatted at 2026-08-10 14:40:46 on dist-test-slave-rmhg"
server_key: "fa7fc2e34dcde34f9988589276e35cec"
server_key_iv: "3bf5e25ad609518586ef3f2190162a31"
server_key_version: "encryptionkey@0"
I20260810 14:40:46.575733 22648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:40:46.578372 22670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:46.579707 22648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:40:46.579840 22648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "6e7fe2aa2a1a432e9bb72e3715a538fa"
format_stamp: "Formatted at 2026-08-10 14:40:46 on dist-test-slave-rmhg"
server_key: "fa7fc2e34dcde34f9988589276e35cec"
server_key_iv: "3bf5e25ad609518586ef3f2190162a31"
server_key_version: "encryptionkey@0"
I20260810 14:40:46.579957 22648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:46.585888 22648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:46.586555 22648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:46.586727 22648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:46.593950 22648 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:46107
I20260810 14:40:46.593959 22743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:46107 every 8 connection(s)
I20260810 14:40:46.594972 22648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:46.598450 22744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:46.599876 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 22648
I20260810 14:40:46.599979 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:46.600250 15696 external_mini_cluster.cc:1428] Setting key d055e8c967e7c965b3a272b85cc976c6
I20260810 14:40:46.605109 22744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa: Bootstrap starting.
I20260810 14:40:46.606281 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.601883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.15.84.62:46107)
0810 14:40:46.602605 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:46.602622 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:46.602659 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:40:46.603365 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:46.603385 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:46.603776 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:46.604114 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:46.604926 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:46.604931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:46.604943 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:46.604955 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:46.605013 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:46.605030 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:46.605200 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:46.605280 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 14:40:46.605533 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":152,"threads_started":1}
I20260810 14:40:46.607626 22744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:46.608407 22744 log.cc:824] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:46.610451 22744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa: No bootstrap required, opened a new log
I20260810 14:40:46.614259 22744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7fe2aa2a1a432e9bb72e3715a538fa" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46107 } }
I20260810 14:40:46.614549 22744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:46.614611 22744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7fe2aa2a1a432e9bb72e3715a538fa, State: Initialized, Role: FOLLOWER
I20260810 14:40:46.615088 22744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7fe2aa2a1a432e9bb72e3715a538fa" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46107 } }
I20260810 14:40:46.615233 22744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:46.615306 22744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:46.615410 22744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:46.616365 22744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7fe2aa2a1a432e9bb72e3715a538fa" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46107 } }
I20260810 14:40:46.616705 22744 leader_election.cc:302] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e7fe2aa2a1a432e9bb72e3715a538fa; no voters:
I20260810 14:40:46.617015 22744 leader_election.cc:288] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:46.617141 22752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:46.617393 22752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [term 1 LEADER]: Becoming Leader. State: Replica: 6e7fe2aa2a1a432e9bb72e3715a538fa, State: Running, Role: LEADER
I20260810 14:40:46.617725 22752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7fe2aa2a1a432e9bb72e3715a538fa" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46107 } }
I20260810 14:40:46.618312 22744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:46.619294 22757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e7fe2aa2a1a432e9bb72e3715a538fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7fe2aa2a1a432e9bb72e3715a538fa" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46107 } } }
I20260810 14:40:46.619422 22757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:46.619707 22758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e7fe2aa2a1a432e9bb72e3715a538fa. Latest consensus state: current_term: 1 leader_uuid: "6e7fe2aa2a1a432e9bb72e3715a538fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7fe2aa2a1a432e9bb72e3715a538fa" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46107 } } }
I20260810 14:40:46.619800 22758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:46.620127 22769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:46.623068 22769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:46.628722 22769 catalog_manager.cc:1347] Generated new cluster ID: 16f07aa91df541ecba08012357581242
I20260810 14:40:46.628829 22769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:46.658219 22769 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:46.658643 22769 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:46.667730 22769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa: Generated new TSK 0
I20260810 14:40:46.668773 22769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:40:46.673223 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:46107
--builtin_ntp_servers=127.15.84.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 14:40:46.795132 22783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:46.795444 22783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:46.795487 22783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:46.799819 22783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:46.799908 22783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:46.799969 22783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:40:46.805075 22783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:46107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:46.806347 22783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:46.807873 22783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:46.818413 22796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:46.818460 22794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:46.818411 22792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:46.818707 22783 server_base.cc:1034] running on GCE node
I20260810 14:40:46.819289 22783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:46.820101 22783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:46.821236 22783 hybrid_clock.cc:648] HybridClock initialized: now 1786372846821211 us; error 38 us; skew 500 ppm
I20260810 14:40:46.823359 22783 webserver.cc:460] Webserver started at http://127.15.84.1:39661/ using document root <none> and password file <none>
I20260810 14:40:46.823956 22783 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:46.824039 22783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:46.824280 22783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:46.826669 22783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "3d8163f7066d4a7a97c3f41ec32df665"
format_stamp: "Formatted at 2026-08-10 14:40:46 on dist-test-slave-rmhg"
server_key: "0cc0adb72ca14475e139de906cf41d98"
server_key_iv: "a88cc9751e18a0afbea7f0795d1b76cb"
server_key_version: "encryptionkey@0"
I20260810 14:40:46.827322 22783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "3d8163f7066d4a7a97c3f41ec32df665"
format_stamp: "Formatted at 2026-08-10 14:40:46 on dist-test-slave-rmhg"
server_key: "0cc0adb72ca14475e139de906cf41d98"
server_key_iv: "a88cc9751e18a0afbea7f0795d1b76cb"
server_key_version: "encryptionkey@0"
I20260810 14:40:46.831566 22783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:40:46.834545 22804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:46.835959 22783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:40:46.836102 22783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "3d8163f7066d4a7a97c3f41ec32df665"
format_stamp: "Formatted at 2026-08-10 14:40:46 on dist-test-slave-rmhg"
server_key: "0cc0adb72ca14475e139de906cf41d98"
server_key_iv: "a88cc9751e18a0afbea7f0795d1b76cb"
server_key_version: "encryptionkey@0"
I20260810 14:40:46.836231 22783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:46.841297 22783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:46.841986 22783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:46.842162 22783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:46.842743 22783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:46.843737 22783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:46.843798 22783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:46.843856 22783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:46.843881 22783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:46.852746 22783 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:40537
I20260810 14:40:46.852792 22986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:40537 every 8 connection(s)
I20260810 14:40:46.853765 22783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:40:46.860152 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.855694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:39019 (local address 127.15.84.62:46107)
0810 14:40:46.855848 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:46.855852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:46.857032 (+ 1180us) server_negotiation.cc:407] Connection header received
0810 14:40:46.858515 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:46.858519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:46.858560 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:46.858639 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:46.859781 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:46.859784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:46.859785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:46.859795 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:46.859813 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:46.859820 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:46.859895 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:46.859988 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 14:40:46.860026 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:40:46.861083 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:46.856133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:46107 (local address 127.15.84.1:39019)
0810 14:40:46.856864 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:46.856908 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:46.858273 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:46.858826 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:46.858833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:46.858930 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:46.859492 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:40:46.859494 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:46.859539 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:40:46.859624 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:46.859884 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:46.859889 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:46.859892 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:40:46.860027 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 14:40:46.860255 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":124,"threads_started":1}
I20260810 14:40:46.861109 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 22783
I20260810 14:40:46.861268 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:40:46.861606 15696 external_mini_cluster.cc:1428] Setting key 26ea879d068b6e5fcb13f4ba46de37b2
I20260810 14:40:46.862475 22988 heartbeater.cc:347] Connected to a master server at 127.15.84.62:46107
I20260810 14:40:46.862749 22988 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:46.863257 22988 heartbeater.cc:511] Master 127.15.84.62:46107 requested a full tablet report, sending...
I20260810 14:40:46.863716 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:46107
--builtin_ntp_servers=127.15.84.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 14:40:46.864874 22697 ts_manager.cc:194] Registered new tserver with Master: 3d8163f7066d4a7a97c3f41ec32df665 (127.15.84.1:40537)
W20260810 14:40:46.983021 22992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:46.983343 22992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:46.983388 22992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:46.987679 22992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:46.987764 22992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:46.987836 22992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:40:46.992863 22992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:46107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:46.994131 22992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:46.995651 22992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:47.005806 22998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:47.005868 22997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:47.005967 22992 server_base.cc:1034] running on GCE node
W20260810 14:40:47.005811 23002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:47.006608 22992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:47.007339 22992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:47.008507 22992 hybrid_clock.cc:648] HybridClock initialized: now 1786372847008482 us; error 43 us; skew 500 ppm
I20260810 14:40:47.010541 22992 webserver.cc:460] Webserver started at http://127.15.84.2:46737/ using document root <none> and password file <none>
I20260810 14:40:47.011140 22992 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:47.011214 22992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:47.011408 22992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:47.013621 22992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "bca736ae6d604307a0c4148645c7d95a"
format_stamp: "Formatted at 2026-08-10 14:40:47 on dist-test-slave-rmhg"
server_key: "0249629c4e8faf7542c2a80dfcd67aa4"
server_key_iv: "cd1c22319b592acf5d9831957217546d"
server_key_version: "encryptionkey@0"
I20260810 14:40:47.014191 22992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "bca736ae6d604307a0c4148645c7d95a"
format_stamp: "Formatted at 2026-08-10 14:40:47 on dist-test-slave-rmhg"
server_key: "0249629c4e8faf7542c2a80dfcd67aa4"
server_key_iv: "cd1c22319b592acf5d9831957217546d"
server_key_version: "encryptionkey@0"
I20260810 14:40:47.018230 22992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:40:47.021095 23011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:47.022423 22992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260810 14:40:47.022550 22992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "bca736ae6d604307a0c4148645c7d95a"
format_stamp: "Formatted at 2026-08-10 14:40:47 on dist-test-slave-rmhg"
server_key: "0249629c4e8faf7542c2a80dfcd67aa4"
server_key_iv: "cd1c22319b592acf5d9831957217546d"
server_key_version: "encryptionkey@0"
I20260810 14:40:47.022665 22992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:47.027593 22992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:47.028312 22992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:47.028491 22992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:47.029074 22992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:47.030120 22992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:47.030189 22992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:47.030256 22992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:47.030279 22992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:47.039345 22992 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:45133
I20260810 14:40:47.039413 23182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:45133 every 8 connection(s)
I20260810 14:40:47.040354 22992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:40:47.041293 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 22992
I20260810 14:40:47.041420 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:40:47.041687 15696 external_mini_cluster.cc:1428] Setting key 286348b664a5855f68e88227d6fc508e
I20260810 14:40:47.043922 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:46107
--builtin_ntp_servers=127.15.84.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 14:40:47.048372 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.043489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:60605 (local address 127.15.84.62:46107)
0810 14:40:47.043632 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:47.043644 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:47.045125 (+ 1481us) server_negotiation.cc:407] Connection header received
0810 14:40:47.046673 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:47.046677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:47.046723 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:47.046817 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:47.047955 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:47.047958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:47.047960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:47.047972 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:47.047995 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:47.048003 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:47.048078 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:47.048217 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 14:40:47.048255 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:40:47.049691 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.044023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:46107 (local address 127.15.84.2:60605)
0810 14:40:47.044945 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:47.045006 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:47.046444 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:47.046960 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:47.046967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:47.047065 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:47.047635 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:40:47.047637 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:47.047703 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:40:47.047798 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:47.048080 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:47.048088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:47.048094 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:40:47.048271 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 14:40:47.048530 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":136,"threads_started":1}
I20260810 14:40:47.050891 23183 heartbeater.cc:347] Connected to a master server at 127.15.84.62:46107
I20260810 14:40:47.051175 23183 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:47.051736 23183 heartbeater.cc:511] Master 127.15.84.62:46107 requested a full tablet report, sending...
I20260810 14:40:47.053028 22697 ts_manager.cc:194] Registered new tserver with Master: bca736ae6d604307a0c4148645c7d95a (127.15.84.2:45133)
W20260810 14:40:47.166723 23187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:47.167076 23187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:47.167138 23187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:47.171577 23187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:47.171658 23187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:47.171738 23187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:40:47.176817 23187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:46107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:47.178068 23187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:47.179505 23187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:47.189607 23193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:47.189661 23196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:47.189661 23194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:47.189864 23187 server_base.cc:1034] running on GCE node
I20260810 14:40:47.190397 23187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:47.191133 23187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:47.192325 23187 hybrid_clock.cc:648] HybridClock initialized: now 1786372847192304 us; error 38 us; skew 500 ppm
I20260810 14:40:47.194399 23187 webserver.cc:460] Webserver started at http://127.15.84.3:44009/ using document root <none> and password file <none>
I20260810 14:40:47.195041 23187 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:47.195199 23187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:47.195467 23187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:47.197779 23187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "836f84633a234a31b02a40d41c030385"
format_stamp: "Formatted at 2026-08-10 14:40:47 on dist-test-slave-rmhg"
server_key: "824cf13f52cfa55a1667fcc1bedd496f"
server_key_iv: "646130db97bd50e80bc3f9059e384872"
server_key_version: "encryptionkey@0"
I20260810 14:40:47.198357 23187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "836f84633a234a31b02a40d41c030385"
format_stamp: "Formatted at 2026-08-10 14:40:47 on dist-test-slave-rmhg"
server_key: "824cf13f52cfa55a1667fcc1bedd496f"
server_key_iv: "646130db97bd50e80bc3f9059e384872"
server_key_version: "encryptionkey@0"
I20260810 14:40:47.202414 23187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 14:40:47.205219 23205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:47.206537 23187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:40:47.206661 23187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "836f84633a234a31b02a40d41c030385"
format_stamp: "Formatted at 2026-08-10 14:40:47 on dist-test-slave-rmhg"
server_key: "824cf13f52cfa55a1667fcc1bedd496f"
server_key_iv: "646130db97bd50e80bc3f9059e384872"
server_key_version: "encryptionkey@0"
I20260810 14:40:47.206780 23187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:47.211808 23187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:47.212481 23187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:47.212648 23187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:47.213243 23187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:47.214262 23187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:47.214311 23187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:47.214363 23187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:47.214392 23187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:47.222945 23187 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:34381
I20260810 14:40:47.222990 23367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:34381 every 8 connection(s)
I20260810 14:40:47.223924 23187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:40:47.230444 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.225864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:60977 (local address 127.15.84.62:46107)
0810 14:40:47.226006 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:47.226009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:47.227371 (+ 1362us) server_negotiation.cc:407] Connection header received
0810 14:40:47.228714 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:47.228717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:47.228767 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:47.228843 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:47.229982 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:47.229986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:47.229988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:47.230010 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:47.230031 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:47.230040 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:47.230123 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:47.230254 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 14:40:47.230302 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:40:47.231312 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.226337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:46107 (local address 127.15.84.3:60977)
0810 14:40:47.227213 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:47.227257 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:47.228478 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:47.229042 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:47.229050 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:47.229147 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:47.229712 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:40:47.229714 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:47.229756 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:40:47.229827 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:47.230114 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:47.230123 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:47.230130 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:40:47.230296 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 14:40:47.230557 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":118,"threads_started":1}
I20260810 14:40:47.232466 23368 heartbeater.cc:347] Connected to a master server at 127.15.84.62:46107
I20260810 14:40:47.232736 23368 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:47.232863 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 23187
I20260810 14:40:47.232976 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:40:47.233240 23368 heartbeater.cc:511] Master 127.15.84.62:46107 requested a full tablet report, sending...
I20260810 14:40:47.233268 15696 external_mini_cluster.cc:1428] Setting key a866db1578e58f703c4dd6eb94f76345
I20260810 14:40:47.234336 22697 ts_manager.cc:194] Registered new tserver with Master: 836f84633a234a31b02a40d41c030385 (127.15.84.3:34381)
I20260810 14:40:47.235437 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:40:47.238826 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.237493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.15.84.62:46107)
0810 14:40:47.237637 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:47.237642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:47.237808 (+ 166us) server_negotiation.cc:407] Connection header received
0810 14:40:47.237898 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:47.237900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:47.237932 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:47.237992 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:47.238533 (+ 541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:47.238536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:47.238538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:47.238548 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:47.238565 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:47.238573 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:47.238639 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:47.238666 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 14:40:47.238697 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:40:47.241617 22694 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:40:47.244659 22694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:47.256989 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.253289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.15.84.2:45133)
0810 14:40:47.254733 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:47.254747 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:47.254835 (+ 88us) server_negotiation.cc:407] Connection header received
0810 14:40:47.254918 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:47.254925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:47.255146 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:47.255295 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:47.256314 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:47.256321 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:47.256335 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:47.256347 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:47.256397 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:47.256439 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:47.256556 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:47.256741 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 14:40:47.256823 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1325,"thread_start_us":126,"threads_started":1}
I20260810 14:40:47.257031 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.253113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:45133 (local address 127.0.0.1:42856)
0810 14:40:47.254722 (+ 1609us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:47.254736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:47.254833 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:47.255342 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:47.255348 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:47.255432 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:47.256039 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:40:47.256041 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:47.256063 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:40:47.256151 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:47.256542 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:47.256553 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:47.256560 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:40:47.256682 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 14:40:47.256915 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":104,"threads_started":1}
I20260810 14:40:47.257419 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.254565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.15.84.1:40537)
0810 14:40:47.255027 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:47.255038 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:47.255059 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:40:47.255120 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:47.255125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:47.255297 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:47.255413 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:47.256716 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:47.256722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:47.256734 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:47.256746 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:47.256789 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:47.256824 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:47.256925 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:47.257197 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 14:40:47.257270 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":156,"threads_started":1}
I20260810 14:40:47.257503 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.253302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:34381 (local address 127.0.0.1:48584)
0810 14:40:47.254295 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:47.254354 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:47.254464 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:47.255484 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:47.255487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:47.255498 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:47.256171 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:40:47.256173 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:47.256182 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:40:47.256202 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:47.257052 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:47.257055 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:47.257058 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:40:47.257362 (+ 304us) client_negotiation.cc:240] Negotiation successful
0810 14:40:47.257395 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":128,"threads_started":1}
I20260810 14:40:47.257645 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.254128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:40537 (local address 127.0.0.1:51404)
0810 14:40:47.254885 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:47.254896 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:47.254958 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:47.256023 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:47.256026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:47.256040 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:47.256543 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:40:47.256546 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:47.256560 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:40:47.256586 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:47.256971 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:47.256974 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:47.256978 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:40:47.257049 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 14:40:47.257088 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":113,"threads_started":1}
I20260810 14:40:47.258227 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.253440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.15.84.3:34381)
0810 14:40:47.255089 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:47.255099 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:47.255119 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:40:47.255162 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:47.255166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:47.255339 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:47.256072 (+ 733us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:47.256869 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:47.256874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:47.256884 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:47.256895 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:47.256930 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:47.256958 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:47.257789 (+ 831us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:47.257991 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 14:40:47.258070 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1544,"thread_start_us":1008,"threads_started":1}
I20260810 14:40:47.260221 23088 tablet_service.cc:1467] Processing CreateTablet for tablet 1584aea5d6c54d44b0ef3626ad2779d4 (DEFAULT_TABLE table=test-table [id=7f0935759a644fdb85936e61765779e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:47.260780 23275 tablet_service.cc:1467] Processing CreateTablet for tablet 1584aea5d6c54d44b0ef3626ad2779d4 (DEFAULT_TABLE table=test-table [id=7f0935759a644fdb85936e61765779e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:47.260780 22876 tablet_service.cc:1467] Processing CreateTablet for tablet 1584aea5d6c54d44b0ef3626ad2779d4 (DEFAULT_TABLE table=test-table [id=7f0935759a644fdb85936e61765779e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:47.261390 23088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1584aea5d6c54d44b0ef3626ad2779d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:47.261874 23275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1584aea5d6c54d44b0ef3626ad2779d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:47.261874 22876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1584aea5d6c54d44b0ef3626ad2779d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:47.268127 23390 tablet_bootstrap.cc:492] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a: Bootstrap starting.
I20260810 14:40:47.269930 23391 tablet_bootstrap.cc:492] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665: Bootstrap starting.
I20260810 14:40:47.271029 23390 tablet_bootstrap.cc:654] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:47.271816 23390 log.cc:824] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:47.271883 23391 tablet_bootstrap.cc:654] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:47.272531 23393 tablet_bootstrap.cc:492] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385: Bootstrap starting.
I20260810 14:40:47.272701 23391 log.cc:824] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:47.274084 23390 tablet_bootstrap.cc:492] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a: No bootstrap required, opened a new log
I20260810 14:40:47.274303 23390 ts_tablet_manager.cc:1397] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260810 14:40:47.274609 23393 tablet_bootstrap.cc:654] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:47.274879 23391 tablet_bootstrap.cc:492] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665: No bootstrap required, opened a new log
I20260810 14:40:47.275076 23391 ts_tablet_manager.cc:1397] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 14:40:47.275369 23393 log.cc:824] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:47.277591 23393 tablet_bootstrap.cc:492] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385: No bootstrap required, opened a new log
I20260810 14:40:47.277791 23393 ts_tablet_manager.cc:1397] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 14:40:47.278796 23390 raft_consensus.cc:348] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca736ae6d604307a0c4148645c7d95a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45133 } } peers { permanent_uuid: "836f84633a234a31b02a40d41c030385" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34381 } } peers { permanent_uuid: "3d8163f7066d4a7a97c3f41ec32df665" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40537 } }
I20260810 14:40:47.279099 23390 raft_consensus.cc:374] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:47.279160 23390 raft_consensus.cc:729] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca736ae6d604307a0c4148645c7d95a, State: Initialized, Role: FOLLOWER
I20260810 14:40:47.279660 23390 consensus_queue.cc:260] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca736ae6d604307a0c4148645c7d95a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45133 } } peers { permanent_uuid: "836f84633a234a31b02a40d41c030385" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34381 } } peers { permanent_uuid: "3d8163f7066d4a7a97c3f41ec32df665" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40537 } }
I20260810 14:40:47.280457 23183 heartbeater.cc:503] Master 127.15.84.62:46107 was elected leader, sending a full tablet report...
I20260810 14:40:47.280774 23390 ts_tablet_manager.cc:1428] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 14:40:47.281486 23393 raft_consensus.cc:348] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca736ae6d604307a0c4148645c7d95a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45133 } } peers { permanent_uuid: "836f84633a234a31b02a40d41c030385" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34381 } } peers { permanent_uuid: "3d8163f7066d4a7a97c3f41ec32df665" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40537 } }
I20260810 14:40:47.281757 23393 raft_consensus.cc:374] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:47.281817 23393 raft_consensus.cc:729] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836f84633a234a31b02a40d41c030385, State: Initialized, Role: FOLLOWER
I20260810 14:40:47.282279 23393 consensus_queue.cc:260] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca736ae6d604307a0c4148645c7d95a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45133 } } peers { permanent_uuid: "836f84633a234a31b02a40d41c030385" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34381 } } peers { permanent_uuid: "3d8163f7066d4a7a97c3f41ec32df665" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40537 } }
I20260810 14:40:47.282660 23391 raft_consensus.cc:348] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca736ae6d604307a0c4148645c7d95a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45133 } } peers { permanent_uuid: "836f84633a234a31b02a40d41c030385" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34381 } } peers { permanent_uuid: "3d8163f7066d4a7a97c3f41ec32df665" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40537 } }
I20260810 14:40:47.282931 23391 raft_consensus.cc:374] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:47.282974 23368 heartbeater.cc:503] Master 127.15.84.62:46107 was elected leader, sending a full tablet report...
I20260810 14:40:47.282982 23391 raft_consensus.cc:729] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8163f7066d4a7a97c3f41ec32df665, State: Initialized, Role: FOLLOWER
I20260810 14:40:47.283329 23393 ts_tablet_manager.cc:1428] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 14:40:47.283438 23391 consensus_queue.cc:260] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca736ae6d604307a0c4148645c7d95a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45133 } } peers { permanent_uuid: "836f84633a234a31b02a40d41c030385" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34381 } } peers { permanent_uuid: "3d8163f7066d4a7a97c3f41ec32df665" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40537 } }
I20260810 14:40:47.284502 23391 ts_tablet_manager.cc:1428] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260810 14:40:47.284829 22988 heartbeater.cc:503] Master 127.15.84.62:46107 was elected leader, sending a full tablet report...
W20260810 14:40:47.291370 23184 tablet.cc:2369] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:40:47.354992 22989 tablet.cc:2369] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:40:47.420388 23403 raft_consensus.cc:482] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:47.420632 23403 raft_consensus.cc:504] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca736ae6d604307a0c4148645c7d95a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45133 } } peers { permanent_uuid: "836f84633a234a31b02a40d41c030385" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34381 } } peers { permanent_uuid: "3d8163f7066d4a7a97c3f41ec32df665" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40537 } }
I20260810 14:40:47.421887 23403 leader_election.cc:288] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bca736ae6d604307a0c4148645c7d95a (127.15.84.2:45133), 3d8163f7066d4a7a97c3f41ec32df665 (127.15.84.1:40537)
I20260810 14:40:47.423707 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.422304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:47815 (local address 127.15.84.1:40537)
0810 14:40:47.422445 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:47.422450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:47.422714 (+ 264us) server_negotiation.cc:407] Connection header received
0810 14:40:47.422773 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:47.422775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:47.422807 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:47.422868 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:47.423352 (+ 484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:47.423356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:47.423358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:47.423369 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:47.423388 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:47.423397 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:47.423474 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:47.423544 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 14:40:47.423577 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:40:47.423717 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.422354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:42397 (local address 127.15.84.2:45133)
0810 14:40:47.422467 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:47.422471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:47.422485 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:40:47.422533 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:47.422535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:47.422570 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:47.422636 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:47.423309 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:47.423313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:47.423316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:47.423332 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:47.423357 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:47.423368 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:47.423465 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:47.423521 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 14:40:47.423566 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:40:47.423710 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.422150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:45133 (local address 127.15.84.3:42397)
0810 14:40:47.422317 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:47.422334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:47.422427 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:47.422683 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:47.422686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:47.422700 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:47.423126 (+ 426us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:40:47.423127 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:47.423145 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:40:47.423176 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:47.423445 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:47.423448 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:47.423451 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:40:47.423523 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 14:40:47.423586 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":17}
I20260810 14:40:47.423730 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.422174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:40537 (local address 127.15.84.3:47815)
0810 14:40:47.422595 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:47.422606 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:47.422709 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:47.422898 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:47.422901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:47.422913 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:47.423215 (+ 302us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:40:47.423217 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:47.423224 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:40:47.423245 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:47.423465 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:47.423468 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:47.423470 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:40:47.423532 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 14:40:47.423573 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":131,"threads_started":1}
I20260810 14:40:47.424345 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1584aea5d6c54d44b0ef3626ad2779d4" candidate_uuid: "836f84633a234a31b02a40d41c030385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8163f7066d4a7a97c3f41ec32df665" is_pre_election: true
I20260810 14:40:47.424732 22919 raft_consensus.cc:2454] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836f84633a234a31b02a40d41c030385 in term 0.
I20260810 14:40:47.424853 23119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1584aea5d6c54d44b0ef3626ad2779d4" candidate_uuid: "836f84633a234a31b02a40d41c030385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bca736ae6d604307a0c4148645c7d95a" is_pre_election: true
I20260810 14:40:47.425184 23119 raft_consensus.cc:2454] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836f84633a234a31b02a40d41c030385 in term 0.
I20260810 14:40:47.425264 23211 leader_election.cc:302] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8163f7066d4a7a97c3f41ec32df665, 836f84633a234a31b02a40d41c030385; no voters:
I20260810 14:40:47.425570 23403 raft_consensus.cc:2781] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:40:47.425654 23403 raft_consensus.cc:482] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:40:47.425706 23403 raft_consensus.cc:3037] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:47.426862 23403 raft_consensus.cc:504] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca736ae6d604307a0c4148645c7d95a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45133 } } peers { permanent_uuid: "836f84633a234a31b02a40d41c030385" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34381 } } peers { permanent_uuid: "3d8163f7066d4a7a97c3f41ec32df665" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40537 } }
I20260810 14:40:47.427227 23403 leader_election.cc:288] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [CANDIDATE]: Term 1 election: Requested vote from peers bca736ae6d604307a0c4148645c7d95a (127.15.84.2:45133), 3d8163f7066d4a7a97c3f41ec32df665 (127.15.84.1:40537)
I20260810 14:40:47.427623 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1584aea5d6c54d44b0ef3626ad2779d4" candidate_uuid: "836f84633a234a31b02a40d41c030385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8163f7066d4a7a97c3f41ec32df665"
I20260810 14:40:47.427749 22919 raft_consensus.cc:3037] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:47.427807 23119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1584aea5d6c54d44b0ef3626ad2779d4" candidate_uuid: "836f84633a234a31b02a40d41c030385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bca736ae6d604307a0c4148645c7d95a"
I20260810 14:40:47.427946 23119 raft_consensus.cc:3037] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:47.428814 22919 raft_consensus.cc:2454] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836f84633a234a31b02a40d41c030385 in term 1.
I20260810 14:40:47.429001 23119 raft_consensus.cc:2454] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836f84633a234a31b02a40d41c030385 in term 1.
I20260810 14:40:47.429126 23211 leader_election.cc:302] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8163f7066d4a7a97c3f41ec32df665, 836f84633a234a31b02a40d41c030385; no voters:
I20260810 14:40:47.429430 23403 raft_consensus.cc:2781] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:47.429651 23403 raft_consensus.cc:686] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [term 1 LEADER]: Becoming Leader. State: Replica: 836f84633a234a31b02a40d41c030385, State: Running, Role: LEADER
I20260810 14:40:47.430009 23403 consensus_queue.cc:237] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca736ae6d604307a0c4148645c7d95a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45133 } } peers { permanent_uuid: "836f84633a234a31b02a40d41c030385" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34381 } } peers { permanent_uuid: "3d8163f7066d4a7a97c3f41ec32df665" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40537 } }
I20260810 14:40:47.433485 22697 catalog_manager.cc:5495] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 reported cstate change: term changed from 0 to 1, leader changed from <none> to 836f84633a234a31b02a40d41c030385 (127.15.84.3). New cstate: current_term: 1 leader_uuid: "836f84633a234a31b02a40d41c030385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca736ae6d604307a0c4148645c7d95a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836f84633a234a31b02a40d41c030385" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8163f7066d4a7a97c3f41ec32df665" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40537 } health_report { overall_health: UNKNOWN } } }
I20260810 14:40:47.449525 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.448041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.15.84.3:34381)
0810 14:40:47.448183 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:47.448187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:47.448202 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:40:47.448294 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:47.448297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:47.448359 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:47.448432 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:47.449062 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:47.449065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:47.449068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:47.449081 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:47.449099 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:47.449107 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:47.449249 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:47.449328 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 14:40:47.449371 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:40:47.455760 22919 raft_consensus.cc:1261] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665 [term 1 FOLLOWER]: Refusing update from remote peer 836f84633a234a31b02a40d41c030385: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:47.455760 23119 raft_consensus.cc:1261] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a [term 1 FOLLOWER]: Refusing update from remote peer 836f84633a234a31b02a40d41c030385: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:47.456564 23409 consensus_queue.cc:1035] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8163f7066d4a7a97c3f41ec32df665" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:47.456763 23403 consensus_queue.cc:1035] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bca736ae6d604307a0c4148645c7d95a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 45133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:47.465850 23416 mvcc.cc:205] Tried to move back new op lower bound from 7316983183171162112 to 7316983183076134912. Current Snapshot: MvccSnapshot[applied={T|T < 7316983183171162112}]
I20260810 14:40:47.466264 23415 mvcc.cc:205] Tried to move back new op lower bound from 7316983183171162112 to 7316983183076134912. Current Snapshot: MvccSnapshot[applied={T|T < 7316983183171162112}]
I20260810 14:40:47.473449 22697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:47.473620 22697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:47.476215 22697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e7fe2aa2a1a432e9bb72e3715a538fa: Sending DeleteTablet for 3 replicas of tablet 1584aea5d6c54d44b0ef3626ad2779d4
I20260810 14:40:47.477072 22876 tablet_service.cc:1514] Processing DeleteTablet for tablet 1584aea5d6c54d44b0ef3626ad2779d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:47 UTC) from {username='slave'} at 127.0.0.1:51404
I20260810 14:40:47.477080 23088 tablet_service.cc:1514] Processing DeleteTablet for tablet 1584aea5d6c54d44b0ef3626ad2779d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:47 UTC) from {username='slave'} at 127.0.0.1:42856
I20260810 14:40:47.477694 23431 tablet_replica.cc:330] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665: stopping tablet replica
I20260810 14:40:47.477682 23430 tablet_replica.cc:330] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a: stopping tablet replica
I20260810 14:40:47.477701 23275 tablet_service.cc:1514] Processing DeleteTablet for tablet 1584aea5d6c54d44b0ef3626ad2779d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:47 UTC) from {username='slave'} at 127.0.0.1:48584
I20260810 14:40:47.478039 23430 raft_consensus.cc:2229] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:47.478039 23431 raft_consensus.cc:2229] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:47.478396 23431 raft_consensus.cc:2258] T 1584aea5d6c54d44b0ef3626ad2779d4 P 3d8163f7066d4a7a97c3f41ec32df665 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:47.478396 23430 raft_consensus.cc:2258] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:47.479118 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 22783
I20260810 14:40:47.482924 23430 ts_tablet_manager.cc:1905] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:47.486119 23430 ts_tablet_manager.cc:1918] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:47.486258 23430 log.cc:1196] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/1584aea5d6c54d44b0ef3626ad2779d4
I20260810 14:40:47.486625 23430 ts_tablet_manager.cc:1939] T 1584aea5d6c54d44b0ef3626ad2779d4 P bca736ae6d604307a0c4148645c7d95a: Deleting consensus metadata
I20260810 14:40:47.478850 23432 tablet_replica.cc:330] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385: stopping tablet replica
I20260810 14:40:47.487274 23432 raft_consensus.cc:2229] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:40:47.487875 23432 raft_consensus.cc:2258] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:47.493561 23432 ts_tablet_manager.cc:1905] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:47.497047 23432 ts_tablet_manager.cc:1918] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:47.497218 23432 log.cc:1196] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/1584aea5d6c54d44b0ef3626ad2779d4
I20260810 14:40:47.497651 23432 ts_tablet_manager.cc:1939] T 1584aea5d6c54d44b0ef3626ad2779d4 P 836f84633a234a31b02a40d41c030385: Deleting consensus metadata
I20260810 14:40:47.499552 22673 catalog_manager.cc:4841] TS bca736ae6d604307a0c4148645c7d95a (127.15.84.2:45133): tablet 1584aea5d6c54d44b0ef3626ad2779d4 (table test-table [id=7f0935759a644fdb85936e61765779e6]) successfully deleted
I20260810 14:40:47.501770 22672 catalog_manager.cc:4841] TS 836f84633a234a31b02a40d41c030385 (127.15.84.3:34381): tablet 1584aea5d6c54d44b0ef3626ad2779d4 (table test-table [id=7f0935759a644fdb85936e61765779e6]) successfully deleted
W20260810 14:40:47.509799 22675 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:40537 (error 108)
I20260810 14:40:47.510664 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 22992
I20260810 14:40:47.511092 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.510730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:40537 (local address 127.0.0.1:51410)
0810 14:40:47.510931 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:47.511002 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260810 14:40:47.533476 22675 catalog_manager.cc:4584] TS 3d8163f7066d4a7a97c3f41ec32df665 (127.15.84.1:40537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1584aea5d6c54d44b0ef3626ad2779d4: Network error: Client connection negotiation failed: client connection to 127.15.84.1:40537: connect: Connection refused (error 111)
I20260810 14:40:47.540203 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 23187
I20260810 14:40:47.557403 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 22648
2026-08-10T14:40:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1177 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 14:40:47.590181 15696 mini_oidc.cc:152] Starting JWKS server
I20260810 14:40:47.592793 15696 webserver.cc:460] Webserver started at https://127.0.0.1:46183/ using document root <none> and password file <none>
I20260810 14:40:47.592967 15696 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 14:40:47.593297 15696 webserver.cc:460] Webserver started at http://127.0.0.1:42361/ using document root <none> and password file <none>
2026-08-10T14:40:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:47Z Disabled control of system clock
I20260810 14:40:47.603813 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:45613
--webserver_interface=127.15.84.62
--webserver_port=0
--builtin_ntp_servers=127.15.84.20:39729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:45613
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46183/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/testchainca.cert with env {}
W20260810 14:40:47.722476 23440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:47.722774 23440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:47.722817 23440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:47.726912 23440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:47.726991 23440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:47.727005 23440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:47.727021 23440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:47.727035 23440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:40:47.727051 23440 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:40:47.727062 23440 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46183/jwks.json
I20260810 14:40:47.732733 23440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:39729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:45613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:45613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46183/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:47.733873 23440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:47.734877 23440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:47.743628 23452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:47.743649 23449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:47.743647 23447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:47.743893 23440 server_base.cc:1034] running on GCE node
I20260810 14:40:47.744613 23440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:47.745922 23440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:47.747093 23440 hybrid_clock.cc:648] HybridClock initialized: now 1786372847747079 us; error 34 us; skew 500 ppm
I20260810 14:40:47.749461 23440 webserver.cc:460] Webserver started at http://127.15.84.62:32877/ using document root <none> and password file <none>
I20260810 14:40:47.750092 23440 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:47.750167 23440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:47.750372 23440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:47.752581 23440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "f0551f939fc444bda59401b638f169cd"
format_stamp: "Formatted at 2026-08-10 14:40:47 on dist-test-slave-rmhg"
server_key: "3522570d2488d19f50a967d4469ab541"
server_key_iv: "0e668dc63c94a55da6b64ffd05dbc5db"
server_key_version: "encryptionkey@0"
I20260810 14:40:47.753233 23440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "f0551f939fc444bda59401b638f169cd"
format_stamp: "Formatted at 2026-08-10 14:40:47 on dist-test-slave-rmhg"
server_key: "3522570d2488d19f50a967d4469ab541"
server_key_iv: "0e668dc63c94a55da6b64ffd05dbc5db"
server_key_version: "encryptionkey@0"
I20260810 14:40:47.757122 23440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 14:40:47.759810 23460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:47.761251 23440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:47.761400 23440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "f0551f939fc444bda59401b638f169cd"
format_stamp: "Formatted at 2026-08-10 14:40:47 on dist-test-slave-rmhg"
server_key: "3522570d2488d19f50a967d4469ab541"
server_key_iv: "0e668dc63c94a55da6b64ffd05dbc5db"
server_key_version: "encryptionkey@0"
I20260810 14:40:47.761523 23440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:47.783406 23440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:47.784219 23440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:47.784413 23440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:47.791550 23440 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:45613
I20260810 14:40:47.791570 23539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:45613 every 8 connection(s)
I20260810 14:40:47.792567 23440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:47.796084 23540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:47.801038 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 23440
I20260810 14:40:47.801214 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:47.801502 15696 external_mini_cluster.cc:1428] Setting key 1f087d270ea2fbb57a834dfe6cb09f6b
I20260810 14:40:47.802239 23540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd: Bootstrap starting.
I20260810 14:40:47.804757 23540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:47.805558 23540 log.cc:824] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:47.807425 23540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd: No bootstrap required, opened a new log
I20260810 14:40:47.810999 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.802967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.15.84.62:45613)
0810 14:40:47.803695 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:47.803711 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:47.803750 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:40:47.804398 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:47.804416 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:47.804868 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:47.805221 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:47.805717 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:47.806696 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:47.807623 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:47.808136 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:47.808304 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:47.809142 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:47.809188 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:47.809207 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:47.809606 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:47.809678 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:47.809702 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:47.809792 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:47.809938 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 14:40:47.810222 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":144,"threads_started":1}
I20260810 14:40:47.811262 23540 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0551f939fc444bda59401b638f169cd" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45613 } }
I20260810 14:40:47.811502 23540 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:47.811569 23540 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0551f939fc444bda59401b638f169cd, State: Initialized, Role: FOLLOWER
I20260810 14:40:47.812045 23540 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0551f939fc444bda59401b638f169cd" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45613 } }
I20260810 14:40:47.812223 23540 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:47.812291 23540 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:47.812388 23540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:47.813560 23540 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0551f939fc444bda59401b638f169cd" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45613 } }
I20260810 14:40:47.813933 23540 leader_election.cc:302] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0551f939fc444bda59401b638f169cd; no voters:
I20260810 14:40:47.814129 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:40:47.814287 23540 leader_election.cc:288] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:47.814391 23550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:47.814558 23550 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [term 1 LEADER]: Becoming Leader. State: Replica: f0551f939fc444bda59401b638f169cd, State: Running, Role: LEADER
I20260810 14:40:47.814877 23550 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0551f939fc444bda59401b638f169cd" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45613 } }
I20260810 14:40:47.815665 23540 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:47.816336 23550 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0551f939fc444bda59401b638f169cd. Latest consensus state: current_term: 1 leader_uuid: "f0551f939fc444bda59401b638f169cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0551f939fc444bda59401b638f169cd" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45613 } } }
I20260810 14:40:47.816461 23550 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:47.816857 23551 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0551f939fc444bda59401b638f169cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0551f939fc444bda59401b638f169cd" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45613 } } }
I20260810 14:40:47.816960 23551 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:47.817271 23559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:47.820055 23559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:47.825930 23559 catalog_manager.cc:1347] Generated new cluster ID: 9320977d2e0f48448c6e6e02008bc43f
I20260810 14:40:47.826032 23559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:47.845005 23559 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:47.846442 23559 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:47.853262 23559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0551f939fc444bda59401b638f169cd: Generated new TSK 0
I20260810 14:40:47.854235 23559 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 14:40:47.888684 23587 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 14:40:47.888805 23587 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 14:40:47.889510 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:47.865668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.15.84.62:45613)
0810 14:40:47.865813 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:47.865818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:47.866052 (+ 234us) server_negotiation.cc:407] Connection header received
0810 14:40:47.866322 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:47.866325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:47.866382 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:47.866474 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:40:47.866764 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:47.867356 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:47.868257 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:47.868499 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:47.868590 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:47.868978 (+ 388us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:40:47.889010 (+ 20032us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 14:40:47.889365 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38514: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 14:40:47.889602 23548 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38514: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 14:40:47.890956 23586 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:47.865518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:45613 (local address 127.0.0.1:38514)
0810 14:40:47.865949 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:47.865966 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:47.866145 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:47.866506 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:47.866510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:47.866519 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:40:47.866650 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:47.866657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:47.867483 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:47.867488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:47.868141 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:47.868150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:47.868619 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:47.868623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:47.868722 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:47.868773 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:40:47.889292 (+ 20519us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 14:40:47.889526 (+ 234us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.84.62:45613: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":256,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.15.84.62:45613: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 14:40:47.891664 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 23440
2026-08-10T14:40:47Z chronyd exiting
I20260810 14:40:47.937776 15696 test_util.cc:175] -----------------------------------------------
I20260810 14:40:47.937871 15696 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372819317171-15696-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (349 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 14:40:47.938936 15696 mini_oidc.cc:152] Starting JWKS server
I20260810 14:40:47.939827 15696 webserver.cc:460] Webserver started at https://127.0.0.1:34143/ using document root <none> and password file <none>
I20260810 14:40:47.939929 15696 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 14:40:47.940198 15696 webserver.cc:460] Webserver started at http://127.0.0.1:38951/ using document root <none> and password file <none>
2026-08-10T14:40:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:47Z Disabled control of system clock
I20260810 14:40:47.950186 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:43215
--webserver_interface=127.15.84.62
--webserver_port=0
--builtin_ntp_servers=127.15.84.20:42583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:43215
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34143/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/testchainca.cert with env {}
W20260810 14:40:48.067694 23595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:48.067982 23595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:48.068023 23595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:48.072008 23595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:48.072090 23595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:48.072104 23595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:48.072120 23595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:48.072140 23595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:40:48.072155 23595 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:40:48.072167 23595 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34143/jwks.json
I20260810 14:40:48.077708 23595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:42583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:43215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:43215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34143/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:48.078904 23595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:48.079907 23595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:48.088840 23608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:48.088851 23601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:48.088893 23595 server_base.cc:1034] running on GCE node
W20260810 14:40:48.088851 23605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:48.089695 23595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:48.090852 23595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:48.092022 23595 hybrid_clock.cc:648] HybridClock initialized: now 1786372848092001 us; error 35 us; skew 500 ppm
I20260810 14:40:48.094045 23595 webserver.cc:460] Webserver started at http://127.15.84.62:35983/ using document root <none> and password file <none>
I20260810 14:40:48.094635 23595 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:48.094717 23595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:48.094924 23595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:48.097057 23595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "d4e0141904e442f297724aab4dbd07d6"
format_stamp: "Formatted at 2026-08-10 14:40:48 on dist-test-slave-rmhg"
server_key: "86ff8b1e7b3c2c193bf096c1271d3708"
server_key_iv: "f463fe8f66e255655a4140c25ac050c0"
server_key_version: "encryptionkey@0"
I20260810 14:40:48.097661 23595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "d4e0141904e442f297724aab4dbd07d6"
format_stamp: "Formatted at 2026-08-10 14:40:48 on dist-test-slave-rmhg"
server_key: "86ff8b1e7b3c2c193bf096c1271d3708"
server_key_iv: "f463fe8f66e255655a4140c25ac050c0"
server_key_version: "encryptionkey@0"
I20260810 14:40:48.101338 23595 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:40:48.103761 23618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:48.105044 23595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:40:48.105201 23595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "d4e0141904e442f297724aab4dbd07d6"
format_stamp: "Formatted at 2026-08-10 14:40:48 on dist-test-slave-rmhg"
server_key: "86ff8b1e7b3c2c193bf096c1271d3708"
server_key_iv: "f463fe8f66e255655a4140c25ac050c0"
server_key_version: "encryptionkey@0"
I20260810 14:40:48.105309 23595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:48.126807 23595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:48.127581 23595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:48.127787 23595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:48.134946 23595 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:43215
I20260810 14:40:48.135020 23696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:43215 every 8 connection(s)
I20260810 14:40:48.135968 23595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:48.136552 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 23595
I20260810 14:40:48.136670 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:48.136936 15696 external_mini_cluster.cc:1428] Setting key acd5a1345116063311dabceb0d371d22
I20260810 14:40:48.139861 23697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:48.146819 23697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6: Bootstrap starting.
I20260810 14:40:48.147498 23704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:48.138351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.15.84.62:43215)
0810 14:40:48.139068 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:48.139088 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:48.139132 (+ 44us) server_negotiation.cc:407] Connection header received
0810 14:40:48.140041 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:48.140067 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:48.140574 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:48.140956 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:48.141427 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.142893 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.143734 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.144408 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.144596 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:48.145399 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:48.145405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:48.145423 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:48.145949 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:48.146022 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:48.146048 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:48.146151 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:48.146224 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 14:40:48.146505 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":133,"threads_started":1}
I20260810 14:40:48.149516 23697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:48.150362 23697 log.cc:824] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:48.150447 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:40:48.152472 23697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6: No bootstrap required, opened a new log
I20260810 14:40:48.156062 23697 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0141904e442f297724aab4dbd07d6" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 43215 } }
I20260810 14:40:48.156328 23697 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:48.156402 23697 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e0141904e442f297724aab4dbd07d6, State: Initialized, Role: FOLLOWER
I20260810 14:40:48.156855 23697 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0141904e442f297724aab4dbd07d6" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 43215 } }
I20260810 14:40:48.156994 23697 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:48.157065 23697 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:48.157199 23697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:48.158320 23697 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0141904e442f297724aab4dbd07d6" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 43215 } }
I20260810 14:40:48.158663 23697 leader_election.cc:302] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4e0141904e442f297724aab4dbd07d6; no voters:
I20260810 14:40:48.158985 23697 leader_election.cc:288] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:48.159104 23706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:48.159320 23706 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [term 1 LEADER]: Becoming Leader. State: Replica: d4e0141904e442f297724aab4dbd07d6, State: Running, Role: LEADER
I20260810 14:40:48.159677 23706 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0141904e442f297724aab4dbd07d6" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 43215 } }
I20260810 14:40:48.160234 23697 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:48.161329 23709 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4e0141904e442f297724aab4dbd07d6. Latest consensus state: current_term: 1 leader_uuid: "d4e0141904e442f297724aab4dbd07d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0141904e442f297724aab4dbd07d6" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 43215 } } }
I20260810 14:40:48.161363 23708 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4e0141904e442f297724aab4dbd07d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e0141904e442f297724aab4dbd07d6" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 43215 } } }
I20260810 14:40:48.161461 23709 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:48.161460 23708 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:48.161924 23720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:48.165141 23720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:48.170964 23720 catalog_manager.cc:1347] Generated new cluster ID: 5892434fa5cc45c793633095681d454c
I20260810 14:40:48.171042 23720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:48.180915 23720 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:48.181960 23720 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:48.191260 23720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4e0141904e442f297724aab4dbd07d6: Generated new TSK 0
I20260810 14:40:48.192078 23720 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 14:40:48.218706 23750 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 14:40:48.218814 23750 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 14:40:48.219506 23704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:48.203276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59718 (local address 127.15.84.62:43215)
0810 14:40:48.203426 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:48.203431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:48.203621 (+ 190us) server_negotiation.cc:407] Connection header received
0810 14:40:48.203800 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:48.203803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:48.203856 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:48.203952 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:40:48.204248 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.204820 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.206124 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.206339 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.206416 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:48.206766 (+ 350us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:40:48.219129 (+ 12363us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 14:40:48.219385 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59718: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 14:40:48.219569 23704 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59718: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 14:40:48.219795 23747 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:48.203176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:43215 (local address 127.0.0.1:59718)
0810 14:40:48.203516 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:48.203531 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:48.203671 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:48.203972 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:48.203976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:48.203985 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:40:48.204113 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:48.204120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.204939 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.204943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:48.205995 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:48.206008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.206454 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.206458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:48.206547 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:48.206580 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:40:48.219366 (+ 12786us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 14:40:48.219588 (+ 222us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.84.62:43215: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260810 14:40:48.220489 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 23595
2026-08-10T14:40:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (334 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 14:40:48.274048 15696 mini_oidc.cc:152] Starting JWKS server
I20260810 14:40:48.274977 15696 webserver.cc:460] Webserver started at https://127.0.0.1:36821/ using document root <none> and password file <none>
I20260810 14:40:48.275089 15696 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 14:40:48.275355 15696 webserver.cc:460] Webserver started at http://127.0.0.1:40733/ using document root <none> and password file <none>
2026-08-10T14:40:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:48Z Disabled control of system clock
I20260810 14:40:48.285393 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:35187
--webserver_interface=127.15.84.62
--webserver_port=0
--builtin_ntp_servers=127.15.84.20:34801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:35187
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36821/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 14:40:48.402547 23757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:48.402837 23757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:48.402882 23757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:48.406955 23757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:48.407025 23757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:48.407040 23757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:48.407069 23757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:48.407097 23757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:40:48.407128 23757 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:40:48.407162 23757 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36821/jwks.json
I20260810 14:40:48.412024 23757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:35187
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:35187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36821/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:48.413143 23757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:48.414583 23757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:48.423043 23768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:48.423199 23757 server_base.cc:1034] running on GCE node
W20260810 14:40:48.423033 23765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:48.423033 23770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:48.423924 23757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:48.425209 23757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:48.426369 23757 hybrid_clock.cc:648] HybridClock initialized: now 1786372848426350 us; error 35 us; skew 500 ppm
I20260810 14:40:48.428586 23757 webserver.cc:460] Webserver started at http://127.15.84.62:41853/ using document root <none> and password file <none>
I20260810 14:40:48.429313 23757 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:48.429391 23757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:48.429620 23757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:48.431938 23757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "7c0901aede424c6db2530f50c09dd127"
format_stamp: "Formatted at 2026-08-10 14:40:48 on dist-test-slave-rmhg"
server_key: "da884a44001f845afd9ffb91603135e4"
server_key_iv: "96c7c2cf2adb164f6bb9a66ac84d6837"
server_key_version: "encryptionkey@0"
I20260810 14:40:48.432559 23757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "7c0901aede424c6db2530f50c09dd127"
format_stamp: "Formatted at 2026-08-10 14:40:48 on dist-test-slave-rmhg"
server_key: "da884a44001f845afd9ffb91603135e4"
server_key_iv: "96c7c2cf2adb164f6bb9a66ac84d6837"
server_key_version: "encryptionkey@0"
I20260810 14:40:48.436678 23757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:40:48.439425 23781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:48.440899 23757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:40:48.441042 23757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "7c0901aede424c6db2530f50c09dd127"
format_stamp: "Formatted at 2026-08-10 14:40:48 on dist-test-slave-rmhg"
server_key: "da884a44001f845afd9ffb91603135e4"
server_key_iv: "96c7c2cf2adb164f6bb9a66ac84d6837"
server_key_version: "encryptionkey@0"
I20260810 14:40:48.441224 23757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:48.460744 23757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:48.461575 23757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:48.461782 23757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:48.468964 23757 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:35187
I20260810 14:40:48.468974 23859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:35187 every 8 connection(s)
I20260810 14:40:48.470037 23757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:48.471611 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 23757
I20260810 14:40:48.471745 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:48.472038 15696 external_mini_cluster.cc:1428] Setting key f0a2606e2a35ae70d7b5d1bb4a1b1fce
I20260810 14:40:48.474180 23860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:48.480311 23860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127: Bootstrap starting.
I20260810 14:40:48.481738 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:48.473526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.15.84.62:35187)
0810 14:40:48.474268 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:48.474291 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:48.474333 (+ 42us) server_negotiation.cc:407] Connection header received
0810 14:40:48.475013 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:48.475038 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:48.475522 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:48.475883 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:48.476351 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.477422 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.478330 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.478817 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.478965 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:48.479892 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:48.479897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:48.479911 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:48.480316 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:48.480375 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:48.480401 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:48.480531 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:48.480669 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 14:40:48.480945 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":162,"threads_started":1}
I20260810 14:40:48.482837 23860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:48.483722 23860 log.cc:824] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:48.484730 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:40:48.486024 23860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127: No bootstrap required, opened a new log
I20260810 14:40:48.489928 23860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0901aede424c6db2530f50c09dd127" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 35187 } }
I20260810 14:40:48.490168 23860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:48.490212 23860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0901aede424c6db2530f50c09dd127, State: Initialized, Role: FOLLOWER
I20260810 14:40:48.490581 23860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0901aede424c6db2530f50c09dd127" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 35187 } }
I20260810 14:40:48.490694 23860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:48.490738 23860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:48.490808 23860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:48.491686 23860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0901aede424c6db2530f50c09dd127" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 35187 } }
I20260810 14:40:48.491995 23860 leader_election.cc:302] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c0901aede424c6db2530f50c09dd127; no voters:
I20260810 14:40:48.492287 23860 leader_election.cc:288] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:48.492383 23866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:48.492583 23866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [term 1 LEADER]: Becoming Leader. State: Replica: 7c0901aede424c6db2530f50c09dd127, State: Running, Role: LEADER
I20260810 14:40:48.492920 23866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0901aede424c6db2530f50c09dd127" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 35187 } }
I20260810 14:40:48.493700 23860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:48.494801 23868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c0901aede424c6db2530f50c09dd127. Latest consensus state: current_term: 1 leader_uuid: "7c0901aede424c6db2530f50c09dd127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0901aede424c6db2530f50c09dd127" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 35187 } } }
I20260810 14:40:48.494948 23868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:48.495332 23877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:48.496311 23867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c0901aede424c6db2530f50c09dd127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0901aede424c6db2530f50c09dd127" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 35187 } } }
I20260810 14:40:48.496414 23867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:48.498515 23877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:48.503840 23877 catalog_manager.cc:1347] Generated new cluster ID: 2fe24f7fe9454824aec4ef8f0f381fe2
I20260810 14:40:48.503909 23877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:48.525852 23877 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:48.526860 23877 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:48.531935 23877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c0901aede424c6db2530f50c09dd127: Generated new TSK 0
I20260810 14:40:48.532667 23877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:40:48.554826 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:48.549537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.15.84.62:35187)
0810 14:40:48.549673 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:48.549677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:48.549805 (+ 128us) server_negotiation.cc:407] Connection header received
0810 14:40:48.549976 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:48.549979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:48.550042 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:48.550163 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:40:48.550363 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.551202 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.552061 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.552412 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.552525 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:48.552797 (+ 272us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:40:48.554417 (+ 1620us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36821/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 14:40:48.554675 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60594: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36821/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 14:40:48.554911 23863 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60594: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36821/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 14:40:48.556376 23900 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:48.549422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:35187 (local address 127.0.0.1:60594)
0810 14:40:48.549721 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:48.549733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:48.549836 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:48.550166 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:48.550168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:48.550174 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:40:48.550274 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:48.550280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.551304 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.551306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:48.551952 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:48.551960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.552522 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.552524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:48.552584 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:48.552608 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:40:48.554600 (+ 1992us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36821/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 14:40:48.556199 (+ 1599us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.84.62:35187: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36821/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":1393,"thread_start_us":94,"threads_started":1}
I20260810 14:40:48.557226 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 23757
E20260810 14:40:48.579144 23901 webserver.cc:533] Webserver: sslize failed:
E20260810 14:40:48.579234 23901 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T14:40:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (336 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T14:40:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:48Z Disabled control of system clock
I20260810 14:40:48.619478 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:39479
--webserver_interface=127.15.84.62
--webserver_port=0
--builtin_ntp_servers=127.15.84.20:42181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:39479
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/testchainca.cert with env {}
W20260810 14:40:48.732913 23908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:48.733239 23908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:48.733282 23908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:48.737268 23908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:48.737330 23908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:48.737344 23908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:48.737358 23908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:48.737371 23908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:40:48.737385 23908 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:40:48.737397 23908 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 14:40:48.742792 23908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:42181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:39479
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:39479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:48.743871 23908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:48.744897 23908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:48.753412 23913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:48.753419 23917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:48.753440 23921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:48.753819 23908 server_base.cc:1034] running on GCE node
I20260810 14:40:48.754420 23908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:48.755530 23908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:48.756716 23908 hybrid_clock.cc:648] HybridClock initialized: now 1786372848756699 us; error 28 us; skew 500 ppm
I20260810 14:40:48.758662 23908 webserver.cc:460] Webserver started at http://127.15.84.62:34747/ using document root <none> and password file <none>
I20260810 14:40:48.759244 23908 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:48.759308 23908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:48.759509 23908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:48.761667 23908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "a84d68fb780e48c09d4c6189cdcec4cc"
format_stamp: "Formatted at 2026-08-10 14:40:48 on dist-test-slave-rmhg"
server_key: "c924a73628606c0e14dfaa63415f4e60"
server_key_iv: "6997ea6040035c1472bb534364d0d336"
server_key_version: "encryptionkey@0"
I20260810 14:40:48.762225 23908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "a84d68fb780e48c09d4c6189cdcec4cc"
format_stamp: "Formatted at 2026-08-10 14:40:48 on dist-test-slave-rmhg"
server_key: "c924a73628606c0e14dfaa63415f4e60"
server_key_iv: "6997ea6040035c1472bb534364d0d336"
server_key_version: "encryptionkey@0"
I20260810 14:40:48.766050 23908 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:40:48.768483 23932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:48.769721 23908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:40:48.769838 23908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "a84d68fb780e48c09d4c6189cdcec4cc"
format_stamp: "Formatted at 2026-08-10 14:40:48 on dist-test-slave-rmhg"
server_key: "c924a73628606c0e14dfaa63415f4e60"
server_key_iv: "6997ea6040035c1472bb534364d0d336"
server_key_version: "encryptionkey@0"
I20260810 14:40:48.769938 23908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:48.779356 23908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:48.780087 23908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:48.780274 23908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:48.787274 23908 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:39479
I20260810 14:40:48.787279 24022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:39479 every 8 connection(s)
I20260810 14:40:48.788260 23908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:48.791435 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:48.795115 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 23908
I20260810 14:40:48.795246 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:48.795544 15696 external_mini_cluster.cc:1428] Setting key e30e8d1c024a46243ef580496b75644a
I20260810 14:40:48.797719 24023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc: Bootstrap starting.
I20260810 14:40:48.800004 24023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:48.800736 24023 log.cc:824] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:48.802623 24023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc: No bootstrap required, opened a new log
I20260810 14:40:48.805223 24027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:48.796972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.15.84.62:39479)
0810 14:40:48.797668 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:48.797689 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:48.797726 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:40:48.798450 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:48.798470 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:48.798889 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:48.799239 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:48.799713 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.800750 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.801709 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.802200 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.802356 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:48.803192 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:48.803198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:48.803211 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:48.803601 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:48.803662 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:48.803683 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:48.803772 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:48.803918 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 14:40:48.804200 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":172,"threads_started":1}
I20260810 14:40:48.805931 24023 raft_consensus.cc:348] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84d68fb780e48c09d4c6189cdcec4cc" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39479 } }
I20260810 14:40:48.806217 24023 raft_consensus.cc:374] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:48.806278 24023 raft_consensus.cc:729] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84d68fb780e48c09d4c6189cdcec4cc, State: Initialized, Role: FOLLOWER
I20260810 14:40:48.806731 24023 consensus_queue.cc:260] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84d68fb780e48c09d4c6189cdcec4cc" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39479 } }
I20260810 14:40:48.806861 24023 raft_consensus.cc:388] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:48.806923 24023 raft_consensus.cc:482] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:48.807016 24023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:48.808099 24023 raft_consensus.cc:504] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84d68fb780e48c09d4c6189cdcec4cc" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39479 } }
I20260810 14:40:48.808195 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:40:48.808445 24023 leader_election.cc:302] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a84d68fb780e48c09d4c6189cdcec4cc; no voters:
I20260810 14:40:48.808737 24023 leader_election.cc:288] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:48.808869 24035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:48.809084 24035 raft_consensus.cc:686] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [term 1 LEADER]: Becoming Leader. State: Replica: a84d68fb780e48c09d4c6189cdcec4cc, State: Running, Role: LEADER
I20260810 14:40:48.809500 24035 consensus_queue.cc:237] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84d68fb780e48c09d4c6189cdcec4cc" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39479 } }
I20260810 14:40:48.810061 24023 sys_catalog.cc:564] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:48.811188 24037 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a84d68fb780e48c09d4c6189cdcec4cc. Latest consensus state: current_term: 1 leader_uuid: "a84d68fb780e48c09d4c6189cdcec4cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84d68fb780e48c09d4c6189cdcec4cc" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39479 } } }
I20260810 14:40:48.811331 24037 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:48.811750 24050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:48.811722 24036 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a84d68fb780e48c09d4c6189cdcec4cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84d68fb780e48c09d4c6189cdcec4cc" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39479 } } }
I20260810 14:40:48.811861 24036 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:48.814685 24050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:48.820427 24050 catalog_manager.cc:1347] Generated new cluster ID: 8bf59154bcf14dff8262e0019723f9e9
I20260810 14:40:48.820518 24050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:48.839504 24050 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:48.840842 24050 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:48.849141 24050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a84d68fb780e48c09d4c6189cdcec4cc: Generated new TSK 0
I20260810 14:40:48.850069 24050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:40:48.903944 24027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:48.859462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.15.84.62:39479)
0810 14:40:48.859604 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:48.859608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:48.859732 (+ 124us) server_negotiation.cc:407] Connection header received
0810 14:40:48.859901 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:48.859905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:48.859967 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:48.860095 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:40:48.860301 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.861201 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.862119 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.862495 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.862596 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:48.862917 (+ 321us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:40:48.903538 (+ 40621us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 14:40:48.903826 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55416: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 14:40:48.904026 24027 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55416: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 14:40:48.904263 24070 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:48.859324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:39479 (local address 127.0.0.1:55416)
0810 14:40:48.859644 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:48.859657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:48.859764 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:48.860088 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:48.860090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:48.860096 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:40:48.860204 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:48.860209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.861330 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.861333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:48.862001 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:48.862011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:48.862619 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:48.862622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:48.862705 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:48.862734 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:40:48.903809 (+ 41075us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 14:40:48.904049 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.84.62:39479: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":105,"threads_started":1}
I20260810 14:40:48.904997 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 23908
2026-08-10T14:40:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (328 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (130 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (131 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:49 dist-test-slave-rmhg krb5kdc[24088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:49 dist-test-slave-rmhg krb5kdc[24088](info): set up 2 sockets
Aug 10 14:40:49 dist-test-slave-rmhg krb5kdc[24088](info): commencing operation
krb5kdc: starting...
W20260810 14:40:50.332988 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:40:50 dist-test-slave-rmhg krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:40:50.459748 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40797
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:36699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:40797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:50.577924 24108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:50.578241 24108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:50.578303 24108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:50.582695 24108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:50.582779 24108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:50.582806 24108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:50.582828 24108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:50.582858 24108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:50.588560 24108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:36699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:40797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:50.589810 24108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:50.590884 24108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:50.599694 24118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:50.599695 24119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:50.599768 24108 server_base.cc:1034] running on GCE node
W20260810 14:40:50.599673 24123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:50.600461 24108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:50.601699 24108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:50.602852 24108 hybrid_clock.cc:648] HybridClock initialized: now 1786372850602832 us; error 41 us; skew 500 ppm
Aug 10 14:40:50 dist-test-slave-rmhg krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372850, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:50.608381 24108 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:40:50.609328 24108 webserver.cc:460] Webserver started at http://127.15.84.62:34799/ using document root <none> and password file <none>
I20260810 14:40:50.609886 24108 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:50.609953 24108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:50.610146 24108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:50.612205 24108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "1df28be98334456d8255d1de341b7981"
format_stamp: "Formatted at 2026-08-10 14:40:50 on dist-test-slave-rmhg"
server_key: "080283309a031fb1bef791c1c34e7e2a"
server_key_iv: "c79f69c123cb415e3ec8e7b47d8dc80d"
server_key_version: "encryptionkey@0"
I20260810 14:40:50.612761 24108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "1df28be98334456d8255d1de341b7981"
format_stamp: "Formatted at 2026-08-10 14:40:50 on dist-test-slave-rmhg"
server_key: "080283309a031fb1bef791c1c34e7e2a"
server_key_iv: "c79f69c123cb415e3ec8e7b47d8dc80d"
server_key_version: "encryptionkey@0"
I20260810 14:40:50.616397 24108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 14:40:50.618903 24134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:50.620124 24108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:40:50.620242 24108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "1df28be98334456d8255d1de341b7981"
format_stamp: "Formatted at 2026-08-10 14:40:50 on dist-test-slave-rmhg"
server_key: "080283309a031fb1bef791c1c34e7e2a"
server_key_iv: "c79f69c123cb415e3ec8e7b47d8dc80d"
server_key_version: "encryptionkey@0"
I20260810 14:40:50.620337 24108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:50.642642 24108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:50.645694 24108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:50.645901 24108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:50.653017 24108 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:40797
I20260810 14:40:50.653033 24219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:40797 every 8 connection(s)
I20260810 14:40:50.654070 24108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:50.656957 24220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:50.657424 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 24108
I20260810 14:40:50.657555 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:50.657874 15696 external_mini_cluster.cc:1428] Setting key 2228a91ab029359b94ddbbebe9645400
I20260810 14:40:50.664328 24220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981: Bootstrap starting.
I20260810 14:40:50.667096 24220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981: Neither blocks nor log segments found. Creating new log.
Aug 10 14:40:50 dist-test-slave-rmhg krb5kdc[24088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:50.667943 24220 log.cc:824] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:50.670200 24220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981: No bootstrap required, opened a new log
I20260810 14:40:50.672266 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:50.659421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47170 (local address 127.15.84.62:40797)
0810 14:40:50.659898 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:50.659913 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:50.659949 (+ 36us) server_negotiation.cc:407] Connection header received
0810 14:40:50.660633 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:50.660650 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:50.660993 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:50.661324 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:50.662223 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:50.663200 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:50.664364 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:50.664900 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:50.665018 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:50.667684 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:50.667690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:50.667708 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:50.667743 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:50.669526 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:50.669998 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:50.670003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:50.670007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:50.670113 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:50.670401 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:50.670404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:50.670405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:50.670806 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:50.670947 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:50.671241 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 14:40:50.671517 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":144,"threads_started":1}
I20260810 14:40:50.674244 24220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df28be98334456d8255d1de341b7981" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40797 } }
I20260810 14:40:50.674530 24220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:50.674608 24220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df28be98334456d8255d1de341b7981, State: Initialized, Role: FOLLOWER
I20260810 14:40:50.675035 24220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df28be98334456d8255d1de341b7981" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40797 } }
I20260810 14:40:50.675168 24220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:50.675243 24220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:50.675343 24220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:50.676268 24220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df28be98334456d8255d1de341b7981" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40797 } }
I20260810 14:40:50.676611 24220 leader_election.cc:302] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1df28be98334456d8255d1de341b7981; no voters:
I20260810 14:40:50.676932 24220 leader_election.cc:288] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:50.677049 24226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:50.677289 24226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [term 1 LEADER]: Becoming Leader. State: Replica: 1df28be98334456d8255d1de341b7981, State: Running, Role: LEADER
I20260810 14:40:50.677677 24226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df28be98334456d8255d1de341b7981" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40797 } }
I20260810 14:40:50.678158 24220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:50.679554 24227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1df28be98334456d8255d1de341b7981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df28be98334456d8255d1de341b7981" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40797 } } }
I20260810 14:40:50.679587 24230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1df28be98334456d8255d1de341b7981. Latest consensus state: current_term: 1 leader_uuid: "1df28be98334456d8255d1de341b7981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df28be98334456d8255d1de341b7981" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40797 } } }
I20260810 14:40:50.679808 24230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:50.679728 24227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981 [sys.catalog]: This master's current role is: LEADER
W20260810 14:40:50.683370 24251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:40:50.683440 24251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:40:50.683511 24244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:50.684201 24244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:50.689438 24244 catalog_manager.cc:1347] Generated new cluster ID: 8fdfa5e6fa1c4cc394e2bb6eccd3cf27
I20260810 14:40:50.689517 24244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:50.717725 24244 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:50.719080 24244 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:50.732805 24244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1df28be98334456d8255d1de341b7981: Generated new TSK 0
I20260810 14:40:50.733870 24244 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:40:50.765964 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40797
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:36699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:50.886894 24257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:50.887194 24257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:50.887243 24257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:50.891397 24257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:50.891474 24257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:50.891542 24257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:40:50.897054 24257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:36699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:50.898339 24257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:50.899451 24257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:40:50.910006 24257 server_base.cc:1034] running on GCE node
W20260810 14:40:50.909960 24266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:50.909956 24267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:50.909956 24270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:50.910661 24257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:50.911402 24257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:50.912634 24257 hybrid_clock.cc:648] HybridClock initialized: now 1786372850912608 us; error 41 us; skew 500 ppm
Aug 10 14:40:50 dist-test-slave-rmhg krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372850, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:50.915771 24257 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:40:50.916814 24257 webserver.cc:460] Webserver started at http://127.15.84.1:45913/ using document root <none> and password file <none>
I20260810 14:40:50.917460 24257 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:50.917544 24257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:50.917773 24257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:50.920049 24257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "2901d9664afc4d5db94d49bdef6f8e21"
format_stamp: "Formatted at 2026-08-10 14:40:50 on dist-test-slave-rmhg"
server_key: "def07fa14dbeab5b4f55f4a0a8a28656"
server_key_iv: "0193b984ca36e587f0fd60c07824920b"
server_key_version: "encryptionkey@0"
I20260810 14:40:50.920671 24257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "2901d9664afc4d5db94d49bdef6f8e21"
format_stamp: "Formatted at 2026-08-10 14:40:50 on dist-test-slave-rmhg"
server_key: "def07fa14dbeab5b4f55f4a0a8a28656"
server_key_iv: "0193b984ca36e587f0fd60c07824920b"
server_key_version: "encryptionkey@0"
I20260810 14:40:50.924847 24257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 14:40:50.927526 24287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:50.928866 24257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:40:50.928978 24257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "2901d9664afc4d5db94d49bdef6f8e21"
format_stamp: "Formatted at 2026-08-10 14:40:50 on dist-test-slave-rmhg"
server_key: "def07fa14dbeab5b4f55f4a0a8a28656"
server_key_iv: "0193b984ca36e587f0fd60c07824920b"
server_key_version: "encryptionkey@0"
I20260810 14:40:50.929080 24257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:50.944989 24257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:50.948056 24257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:50.948232 24257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:50.948830 24257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:50.949779 24257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:50.949831 24257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:50.949872 24257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:50.949892 24257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:50.959240 24257 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:46487
I20260810 14:40:50.959267 24467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:46487 every 8 connection(s)
I20260810 14:40:50.960264 24257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:40:50.964589 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 24257
I20260810 14:40:50.964712 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:40:50.965031 15696 external_mini_cluster.cc:1428] Setting key f4da558b67948171657fde8a8288ac7c
Aug 10 14:40:50 dist-test-slave-rmhg krb5kdc[24088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372850, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:50.975879 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:50.962320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:37755 (local address 127.15.84.62:40797)
0810 14:40:50.962455 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:50.962459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:50.963394 (+ 935us) server_negotiation.cc:407] Connection header received
0810 14:40:50.964814 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:50.964818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:50.964882 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:50.964979 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:50.966608 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:50.967370 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:50.968620 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:50.968937 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:50.969023 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:50.972172 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:50.972176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:50.972180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:50.972214 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:50.973581 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:50.974207 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:50.974210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:50.974211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:50.974280 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:50.974645 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:50.974647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:50.974648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:50.974848 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:50.974921 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:50.975608 (+ 687us) server_negotiation.cc:299] Negotiation successful
0810 14:40:50.975748 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:40:50.977124 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:50.962724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40797 (local address 127.15.84.1:37755)
0810 14:40:50.963219 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:50.963267 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:50.964558 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:50.965210 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:50.965218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:50.965717 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:50.966446 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:50.966471 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:50.967502 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:50.967505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:50.968482 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:50.968493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:50.969107 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:50.969111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:50.969380 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:50.970076 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:50.970147 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:50.971964 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:50.973702 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:50.973710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:50.973731 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:50.974103 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:50.974359 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:50.974363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:50.974364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:50.974560 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:50.974951 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:50.974959 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:50.975345 (+ 386us) client_negotiation.cc:769] Sending connection context
0810 14:40:50.975595 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 14:40:50.976006 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":129,"threads_started":1}
I20260810 14:40:50.978489 24468 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40797
I20260810 14:40:50.978793 24468 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:50.979475 24468 heartbeater.cc:511] Master 127.15.84.62:40797 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
I20260810 14:40:50.981338 24157 ts_manager.cc:194] Registered new tserver with Master: 2901d9664afc4d5db94d49bdef6f8e21 (127.15.84.1:46487)
I20260810 14:40:50.982753 24157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:37755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:40:50.996373 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40797
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:36699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:51.114707 24476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:51.115012 24476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:51.115059 24476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:51.119343 24476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:51.119424 24476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:51.119498 24476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:40:51.125334 24476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:36699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:51.126458 24476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:51.127473 24476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:40:51.137414 24476 server_base.cc:1034] running on GCE node
W20260810 14:40:51.137383 24487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:51.137377 24485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:51.137364 24489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:51.138142 24476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:51.138819 24476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:51.139976 24476 hybrid_clock.cc:648] HybridClock initialized: now 1786372851139946 us; error 44 us; skew 500 ppm
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372851, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:51.142838 24476 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:40:51.143821 24476 webserver.cc:460] Webserver started at http://127.15.84.2:46255/ using document root <none> and password file <none>
I20260810 14:40:51.144402 24476 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:51.144485 24476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:51.144711 24476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:51.146942 24476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "ab1b1688d3cd46169c7ad6c030d1ee58"
format_stamp: "Formatted at 2026-08-10 14:40:51 on dist-test-slave-rmhg"
server_key: "8b61cd6bdc086d5f8ca889f49c3825b6"
server_key_iv: "6f0ec5f4be86f4cfb0f695d30336d0e4"
server_key_version: "encryptionkey@0"
I20260810 14:40:51.147514 24476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "ab1b1688d3cd46169c7ad6c030d1ee58"
format_stamp: "Formatted at 2026-08-10 14:40:51 on dist-test-slave-rmhg"
server_key: "8b61cd6bdc086d5f8ca889f49c3825b6"
server_key_iv: "6f0ec5f4be86f4cfb0f695d30336d0e4"
server_key_version: "encryptionkey@0"
I20260810 14:40:51.151643 24476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 14:40:51.154172 24499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:51.155462 24476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:40:51.155597 24476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "ab1b1688d3cd46169c7ad6c030d1ee58"
format_stamp: "Formatted at 2026-08-10 14:40:51 on dist-test-slave-rmhg"
server_key: "8b61cd6bdc086d5f8ca889f49c3825b6"
server_key_iv: "6f0ec5f4be86f4cfb0f695d30336d0e4"
server_key_version: "encryptionkey@0"
I20260810 14:40:51.155707 24476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:51.183835 24476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:51.187281 24476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:51.187479 24476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:51.188098 24476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:51.188987 24476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:51.189038 24476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:51.189096 24476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:51.189122 24476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:51.198693 24476 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:41989
I20260810 14:40:51.198730 24672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:41989 every 8 connection(s)
I20260810 14:40:51.199710 24476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:40:51.204119 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 24476
I20260810 14:40:51.204219 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:40:51.204457 15696 external_mini_cluster.cc:1428] Setting key a14be741f6224775a682a3deb6120f9c
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372851, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
I20260810 14:40:51.214970 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:51.201674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:36857 (local address 127.15.84.62:40797)
0810 14:40:51.201811 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:51.201815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:51.202807 (+ 992us) server_negotiation.cc:407] Connection header received
0810 14:40:51.204326 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:51.204330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:51.204390 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:51.204479 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:51.206187 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:51.206728 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:51.207950 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:51.208194 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:51.208285 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:51.211216 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:51.211220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:51.211223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:51.211254 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:51.212649 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:51.213300 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:51.213303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:51.213304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:51.213379 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:51.213765 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:51.213767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:51.213769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:51.213959 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:51.214031 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:51.214721 (+ 690us) server_negotiation.cc:299] Negotiation successful
0810 14:40:51.214832 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:40:51.215971 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:51.202085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40797 (local address 127.15.84.2:36857)
0810 14:40:51.202618 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:51.202670 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:51.204023 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:51.204709 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:51.204717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:51.205258 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:51.206025 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:51.206050 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:51.206861 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:51.206864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:51.207816 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:51.207827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:51.208350 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:51.208354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:51.208544 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:51.209180 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:51.209205 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:51.210976 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:51.212786 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:51.212794 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:51.212812 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:51.213207 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:51.213466 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:51.213469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:51.213471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:51.213666 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:51.214059 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:51.214067 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:51.214457 (+ 390us) client_negotiation.cc:769] Sending connection context
0810 14:40:51.214696 (+ 239us) client_negotiation.cc:240] Negotiation successful
0810 14:40:51.215081 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":115,"threads_started":1}
I20260810 14:40:51.217124 24673 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
I20260810 14:40:51.217429 24673 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:51.218024 24673 heartbeater.cc:511] Master 127.15.84.62:40797 requested a full tablet report, sending...
I20260810 14:40:51.219238 24157 ts_manager.cc:194] Registered new tserver with Master: ab1b1688d3cd46169c7ad6c030d1ee58 (127.15.84.2:41989)
I20260810 14:40:51.219827 24157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:36857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:40:51.232928 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40797
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:36699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:51.350594 24682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:51.350910 24682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:51.350957 24682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:51.355229 24682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:51.355367 24682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:51.355473 24682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:40:51.361339 24682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:36699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:51.362524 24682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:51.363577 24682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:51.373920 24690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:51.374125 24682 server_base.cc:1034] running on GCE node
W20260810 14:40:51.374038 24689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:51.373920 24693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:51.374698 24682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:51.375377 24682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:51.376535 24682 hybrid_clock.cc:648] HybridClock initialized: now 1786372851376507 us; error 43 us; skew 500 ppm
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372851, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:51.379417 24682 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:40:51.380407 24682 webserver.cc:460] Webserver started at http://127.15.84.3:36529/ using document root <none> and password file <none>
I20260810 14:40:51.380976 24682 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:51.381048 24682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:51.381309 24682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:51.383421 24682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "7fabd138028e4e1fbfa859fc5913dc70"
format_stamp: "Formatted at 2026-08-10 14:40:51 on dist-test-slave-rmhg"
server_key: "ded95aabf565e345ac9c17388c7178ec"
server_key_iv: "8c55848437a0428cf473ce478589c8e2"
server_key_version: "encryptionkey@0"
I20260810 14:40:51.383978 24682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "7fabd138028e4e1fbfa859fc5913dc70"
format_stamp: "Formatted at 2026-08-10 14:40:51 on dist-test-slave-rmhg"
server_key: "ded95aabf565e345ac9c17388c7178ec"
server_key_iv: "8c55848437a0428cf473ce478589c8e2"
server_key_version: "encryptionkey@0"
I20260810 14:40:51.387766 24682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:40:51.390244 24702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:51.391438 24682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:40:51.391561 24682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "7fabd138028e4e1fbfa859fc5913dc70"
format_stamp: "Formatted at 2026-08-10 14:40:51 on dist-test-slave-rmhg"
server_key: "ded95aabf565e345ac9c17388c7178ec"
server_key_iv: "8c55848437a0428cf473ce478589c8e2"
server_key_version: "encryptionkey@0"
I20260810 14:40:51.391659 24682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:51.429843 24682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:51.433210 24682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:51.433410 24682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:51.434075 24682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:51.435016 24682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:51.435070 24682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:51.435128 24682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:51.435155 24682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:51.444041 24682 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:37241
I20260810 14:40:51.444068 24880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:37241 every 8 connection(s)
I20260810 14:40:51.444998 24682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:40:51.451066 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 24682
I20260810 14:40:51.451174 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:40:51.451434 15696 external_mini_cluster.cc:1428] Setting key f4f37081df4fc96f86b63d12a65b52c6
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372851, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:40:51.459723 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:51.446930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:34673 (local address 127.15.84.62:40797)
0810 14:40:51.447102 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:51.447106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:51.448002 (+ 896us) server_negotiation.cc:407] Connection header received
0810 14:40:51.449381 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:51.449384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:51.449434 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:51.449510 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:51.450929 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:51.451618 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:51.453040 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:51.453340 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:51.453434 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:51.456115 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:51.456119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:51.456121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:51.456150 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:51.457488 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:51.458033 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:51.458037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:51.458038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:51.458119 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:51.458457 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:51.458461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:51.458463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:51.458681 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:51.458774 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:51.459401 (+ 627us) server_negotiation.cc:299] Negotiation successful
0810 14:40:51.459550 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 14:40:51.460668 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:51.447314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40797 (local address 127.15.84.3:34673)
0810 14:40:51.447827 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:51.447874 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:51.449133 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:51.449713 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:51.449720 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:51.450103 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:51.450725 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:51.450744 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:51.451779 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:51.451782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:51.452908 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:51.452920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:51.453526 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:51.453531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:51.453719 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:51.454302 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:51.454320 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:51.455919 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:51.457617 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:51.457624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:51.457644 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:51.457924 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:51.458207 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:51.458210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:51.458212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:51.458364 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:51.458789 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:51.458797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:51.459148 (+ 351us) client_negotiation.cc:769] Sending connection context
0810 14:40:51.459341 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 14:40:51.459730 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":128,"threads_started":1}
I20260810 14:40:51.461884 24882 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40797
I20260810 14:40:51.462142 24882 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:51.462648 24882 heartbeater.cc:511] Master 127.15.84.62:40797 requested a full tablet report, sending...
I20260810 14:40:51.463702 24157 ts_manager.cc:194] Registered new tserver with Master: 7fabd138028e4e1fbfa859fc5913dc70 (127.15.84.3:37241)
I20260810 14:40:51.464287 24157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:34673
I20260810 14:40:51.465814 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/test-admin.keytab.
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:51.482460 15696 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 14:40:51.485633 24893 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:51.483952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.15.84.1:46487)
0810 14:40:51.484280 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:51.484286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:51.484311 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:40:51.484365 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:51.484370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:51.484578 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:51.484724 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:51.485451 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48792: BlockingRecv error: recv got EOF from 127.0.0.1:48792 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
W20260810 14:40:51.485713 15696 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/krb5cc' of type FILE
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:51.486778 15696 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:51.487605 15696 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.1@KRBTEST.COM
I20260810 14:40:51.497200 24893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:51.488499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.15.84.1:46487)
0810 14:40:51.488623 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:51.488627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:51.488643 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:40:51.488800 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:51.488804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:51.488850 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:51.488918 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:51.489518 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:51.490293 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:51.491134 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:51.491539 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:51.491639 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:51.493850 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:51.493861 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:51.493875 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:51.493901 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:51.495342 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:51.495760 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:51.495766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:51.495767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:51.495855 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:51.496110 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:51.496114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:51.496116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:51.496401 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:51.496602 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:51.496839 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 14:40:51.497006 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 14:40:51.501120 24893 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:51.500122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.15.84.1:46487)
0810 14:40:51.500255 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:51.500259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:51.500334 (+ 75us) server_negotiation.cc:407] Connection header received
0810 14:40:51.500456 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:51.500459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:51.500502 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:51.500567 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:51.501008 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48812: BlockingRecv error: recv got EOF from 127.0.0.1:48812 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 14:40:51.501231 15696 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/krb5cc' of type FILE
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:51.502259 15696 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:51.503049 15696 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.1@KRBTEST.COM
I20260810 14:40:51.511307 24893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:51.503910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48824 (local address 127.15.84.1:46487)
0810 14:40:51.504036 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:51.504040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:51.504116 (+ 76us) server_negotiation.cc:407] Connection header received
0810 14:40:51.504216 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:51.504219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:51.504262 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:51.504328 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:51.504783 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:51.505381 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:51.506140 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:51.506355 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:51.506423 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:51.508623 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:51.508627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:51.508629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:51.508666 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:51.509906 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:51.510305 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:51.510307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:51.510308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:51.510377 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:51.510556 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:51.510558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:51.510559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:51.510754 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:51.510876 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:51.511028 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 14:40:51.511137 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 14:40:51.514055 24893 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:51.513125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.15.84.1:46487)
0810 14:40:51.513291 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:51.513295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:51.513357 (+ 62us) server_negotiation.cc:407] Connection header received
0810 14:40:51.513475 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:51.513477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:51.513522 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:51.513586 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:51.513943 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48840: BlockingRecv error: recv got EOF from 127.0.0.1:48840 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260810 14:40:51.514183 15696 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372819317171-15696-0/krb5kdc/krb5cc' of type FILE
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:51.515178 15696 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:51.515951 15696 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.1@KRBTEST.COM
I20260810 14:40:51.523756 24893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:51.516843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48846 (local address 127.15.84.1:46487)
0810 14:40:51.516964 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:51.516967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:51.517001 (+ 34us) server_negotiation.cc:407] Connection header received
0810 14:40:51.517101 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:51.517104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:51.517141 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:51.517230 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:51.517708 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:51.518212 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:51.518939 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:51.519140 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:51.519206 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:51.521276 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:51.521279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:51.521281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:51.521307 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:51.522393 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:51.522779 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:51.522781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:51.522782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:51.522853 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:51.523046 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:51.523048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:51.523050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:51.523222 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:51.523339 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:51.523484 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 14:40:51.523580 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:40:51.524787 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 24257
I20260810 14:40:51.538915 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 24476
I20260810 14:40:51.559119 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 24682
I20260810 14:40:51.580986 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 24108
2026-08-10T14:40:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2416 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24910](info): set up 2 sockets
Aug 10 14:40:51 dist-test-slave-rmhg krb5kdc[24910](info): commencing operation
krb5kdc: starting...
W20260810 14:40:52.757997 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:40:52 dist-test-slave-rmhg krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.62.keytab.
Entry for principal oryx/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.62.keytab.
I20260810 14:40:52.890228 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:36385
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.62.keytab
--principal=oryx/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:36385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:53.006489 24927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:53.006794 24927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:53.006840 24927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:53.010972 24927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:53.011041 24927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:53.011061 24927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:53.011092 24927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:53.011119 24927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:53.016611 24927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:36385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.62.keytab
--principal=oryx/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:36385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:53.017776 24927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:53.018857 24927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:53.027628 24937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:53.027611 24939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:53.027617 24936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:53.027693 24927 server_base.cc:1034] running on GCE node
I20260810 14:40:53.028452 24927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:53.029640 24927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:53.030802 24927 hybrid_clock.cc:648] HybridClock initialized: now 1786372853030787 us; error 28 us; skew 500 ppm
Aug 10 14:40:53 dist-test-slave-rmhg krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372853, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:53.033601 24927 init.cc:377] Logged in from keytab as oryx/127.15.84.62@KRBTEST.COM (short username oryx)
I20260810 14:40:53.034564 24927 webserver.cc:460] Webserver started at http://127.15.84.62:35741/ using document root <none> and password file <none>
I20260810 14:40:53.035147 24927 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:53.035218 24927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:53.035421 24927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:53.037761 24927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "2767b706cceb48a5bf5440ff912fdea8"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "37f1c44580af9f93a2ad06e346330dfe"
server_key_iv: "7539b3d7c1d2502922b61daf63822f33"
server_key_version: "encryptionkey@0"
I20260810 14:40:53.038442 24927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "2767b706cceb48a5bf5440ff912fdea8"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "37f1c44580af9f93a2ad06e346330dfe"
server_key_iv: "7539b3d7c1d2502922b61daf63822f33"
server_key_version: "encryptionkey@0"
I20260810 14:40:53.042519 24927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:40:53.045277 24952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:53.046594 24927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:40:53.046742 24927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "2767b706cceb48a5bf5440ff912fdea8"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "37f1c44580af9f93a2ad06e346330dfe"
server_key_iv: "7539b3d7c1d2502922b61daf63822f33"
server_key_version: "encryptionkey@0"
I20260810 14:40:53.046855 24927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:53.064016 24927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:53.067080 24927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:53.067268 24927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:53.074529 24927 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:36385
I20260810 14:40:53.074829 25031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:36385 every 8 connection(s)
I20260810 14:40:53.075611 24927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:53.077725 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 24927
I20260810 14:40:53.077868 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:53.078195 15696 external_mini_cluster.cc:1428] Setting key 1ddbee6faa85b5b988872cc96c1927d4
I20260810 14:40:53.079067 25032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:53.086090 25032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8: Bootstrap starting.
Aug 10 14:40:53 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
I20260810 14:40:53.088897 25032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:53.089679 25032 log.cc:824] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:53.091991 25032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8: No bootstrap required, opened a new log
I20260810 14:40:53.092830 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:53.079691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.15.84.62:36385)
0810 14:40:53.080116 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:53.080133 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:53.080174 (+ 41us) server_negotiation.cc:407] Connection header received
0810 14:40:53.080851 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:53.080869 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:53.081266 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:53.081576 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:53.082516 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:53.083558 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:53.084740 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:53.085282 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:53.085402 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:53.088180 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:53.088187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:53.088201 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:53.088234 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:53.089981 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:53.090428 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:53.090434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:53.090441 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:53.090565 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:53.090847 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:53.090849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:53.090850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:53.091264 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:53.091401 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:53.091732 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 14:40:53.092032 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260810 14:40:53.095985 25032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } }
I20260810 14:40:53.096233 25032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:53.096297 25032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2767b706cceb48a5bf5440ff912fdea8, State: Initialized, Role: FOLLOWER
I20260810 14:40:53.096728 25032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } }
I20260810 14:40:53.096863 25032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:53.096926 25032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:53.097030 25032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:53.098026 25032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } }
I20260810 14:40:53.098381 25032 leader_election.cc:302] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2767b706cceb48a5bf5440ff912fdea8; no voters:
I20260810 14:40:53.098724 25032 leader_election.cc:288] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:53.098852 25039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:53.099097 25039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [term 1 LEADER]: Becoming Leader. State: Replica: 2767b706cceb48a5bf5440ff912fdea8, State: Running, Role: LEADER
I20260810 14:40:53.099454 25039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } }
I20260810 14:40:53.100025 25032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:53.101341 25040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } }
I20260810 14:40:53.101495 25040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:53.101311 25043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2767b706cceb48a5bf5440ff912fdea8. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } }
I20260810 14:40:53.101624 25043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:53.101972 25051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:53.105805 25051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:53.111956 25051 catalog_manager.cc:1347] Generated new cluster ID: 53b69ac8f06e4caaaecacf0a13e1c527
I20260810 14:40:53.112051 25051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:53.140902 25051 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:53.142014 25051 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:53.160696 25051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8: Generated new TSK 0
I20260810 14:40:53.161540 25051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.1.keytab.
Entry for principal oryx/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.1.keytab.
I20260810 14:40:53.200204 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:36385
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.1.keytab
--principal=oryx/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:53.342587 25069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:53.342906 25069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:53.342952 25069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:53.347164 25069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:53.347255 25069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:53.347316 25069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:40:53.353477 25069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.1.keytab
--principal=oryx/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:53.354669 25069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:53.355801 25069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:53.365940 25076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:53.366000 25079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:53.366164 25069 server_base.cc:1034] running on GCE node
W20260810 14:40:53.366000 25077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:53.366760 25069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:53.367460 25069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:53.368625 25069 hybrid_clock.cc:648] HybridClock initialized: now 1786372853368604 us; error 34 us; skew 500 ppm
Aug 10 14:40:53 dist-test-slave-rmhg krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372853, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:53.371493 25069 init.cc:377] Logged in from keytab as oryx/127.15.84.1@KRBTEST.COM (short username oryx)
I20260810 14:40:53.372439 25069 webserver.cc:460] Webserver started at http://127.15.84.1:40147/ using document root <none> and password file <none>
I20260810 14:40:53.373001 25069 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:53.373070 25069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:53.373318 25069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:53.375517 25069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "aa6e95f155e54ee58454b5767f2ba60d"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "e0e299db0f37b5029446a399cd23cfcd"
server_key_iv: "4707a2fac06e4113cf43f338a74eb954"
server_key_version: "encryptionkey@0"
I20260810 14:40:53.376071 25069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "aa6e95f155e54ee58454b5767f2ba60d"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "e0e299db0f37b5029446a399cd23cfcd"
server_key_iv: "4707a2fac06e4113cf43f338a74eb954"
server_key_version: "encryptionkey@0"
I20260810 14:40:53.379804 25069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 14:40:53.382225 25092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:53.383394 25069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:40:53.383503 25069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "aa6e95f155e54ee58454b5767f2ba60d"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "e0e299db0f37b5029446a399cd23cfcd"
server_key_iv: "4707a2fac06e4113cf43f338a74eb954"
server_key_version: "encryptionkey@0"
I20260810 14:40:53.383594 25069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:53.418056 25069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:53.421310 25069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:53.421514 25069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:53.422158 25069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:53.423053 25069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:53.423101 25069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:53.423156 25069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:53.423184 25069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:53.432386 25069 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:33165
I20260810 14:40:53.432416 25269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:33165 every 8 connection(s)
I20260810 14:40:53.433470 25069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:40:53.439757 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 25069
I20260810 14:40:53.439893 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:40:53.440171 15696 external_mini_cluster.cc:1428] Setting key cac8b3f1251d9f28be6c89b3e709e5e7
Aug 10 14:40:53 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372853, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
I20260810 14:40:53.448060 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:53.435421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:32959 (local address 127.15.84.62:36385)
0810 14:40:53.435564 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:53.435568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:53.436591 (+ 1023us) server_negotiation.cc:407] Connection header received
0810 14:40:53.438063 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:53.438066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:53.438131 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:53.438213 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:53.439528 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:53.440113 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:53.441407 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:53.441648 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:53.441742 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:53.444271 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:53.444274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:53.444276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:53.444302 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:53.445685 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:53.446323 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:53.446326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:53.446327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:53.446422 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:53.446838 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:53.446841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:53.446842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:53.447054 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:53.447136 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:53.447799 (+ 663us) server_negotiation.cc:299] Negotiation successful
0810 14:40:53.447920 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:40:53.449373 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:53.435842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.1:32959)
0810 14:40:53.436407 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:53.436457 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:53.437822 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:53.438419 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:53.438427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:53.438833 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:53.439353 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:53.439368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:53.440252 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:53.440257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:53.441282 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:53.441292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:53.441804 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:53.441809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:53.442001 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:53.442583 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:53.442601 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:53.444085 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:53.445814 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:53.445820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:53.445837 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:53.446195 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:53.446529 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:53.446532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:53.446534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:53.446739 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:53.447167 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:53.447175 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:53.447541 (+ 366us) client_negotiation.cc:769] Sending connection context
0810 14:40:53.447778 (+ 237us) client_negotiation.cc:240] Negotiation successful
0810 14:40:53.448180 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":131,"threads_started":1}
I20260810 14:40:53.450742 25270 heartbeater.cc:347] Connected to a master server at 127.15.84.62:36385
WARNING: no policy specified for oryx/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:40:53.451047 25270 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:53.451664 25270 heartbeater.cc:511] Master 127.15.84.62:36385 requested a full tablet report, sending...
I20260810 14:40:53.453320 24976 ts_manager.cc:194] Registered new tserver with Master: aa6e95f155e54ee58454b5767f2ba60d (127.15.84.1:33165)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.84.2@KRBTEST.COM" created.
I20260810 14:40:53.454757 24976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:32959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.2.keytab.
Entry for principal oryx/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.2.keytab.
I20260810 14:40:53.469439 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:36385
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.2.keytab
--principal=oryx/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:53.589116 25277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:53.589448 25277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:53.589500 25277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:53.593844 25277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:53.593921 25277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:53.593993 25277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:40:53.599684 25277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.2.keytab
--principal=oryx/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:53.600894 25277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:53.602100 25277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:53.612085 25288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:53.612082 25283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:53.612082 25285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:53.612141 25277 server_base.cc:1034] running on GCE node
I20260810 14:40:53.612924 25277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:53.613642 25277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:53.614813 25277 hybrid_clock.cc:648] HybridClock initialized: now 1786372853614794 us; error 33 us; skew 500 ppm
Aug 10 14:40:53 dist-test-slave-rmhg krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372853, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:53.617630 25277 init.cc:377] Logged in from keytab as oryx/127.15.84.2@KRBTEST.COM (short username oryx)
I20260810 14:40:53.618552 25277 webserver.cc:460] Webserver started at http://127.15.84.2:46245/ using document root <none> and password file <none>
I20260810 14:40:53.619122 25277 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:53.619194 25277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:53.619452 25277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:53.621606 25277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "f2561cc3377241c0a83657629ac58389"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "9483c4bf5a7a4d6a4427bacc3ea908e9"
server_key_iv: "97def4971b81e2e2f4e726897f08aee2"
server_key_version: "encryptionkey@0"
I20260810 14:40:53.622167 25277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "f2561cc3377241c0a83657629ac58389"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "9483c4bf5a7a4d6a4427bacc3ea908e9"
server_key_iv: "97def4971b81e2e2f4e726897f08aee2"
server_key_version: "encryptionkey@0"
I20260810 14:40:53.625968 25277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:40:53.628463 25302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:53.629832 25277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:40:53.629959 25277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "f2561cc3377241c0a83657629ac58389"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "9483c4bf5a7a4d6a4427bacc3ea908e9"
server_key_iv: "97def4971b81e2e2f4e726897f08aee2"
server_key_version: "encryptionkey@0"
I20260810 14:40:53.630070 25277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:53.646116 25277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:53.649052 25277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:53.649281 25277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:53.649943 25277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:53.650827 25277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:53.650878 25277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:53.650933 25277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:53.650997 25277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:53.659646 25277 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:35725
I20260810 14:40:53.659662 25480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:35725 every 8 connection(s)
I20260810 14:40:53.660539 25277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:40:53.666966 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 25277
I20260810 14:40:53.667080 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:40:53.667361 15696 external_mini_cluster.cc:1428] Setting key bea9ee95705067406e0d90e6148322c3
Aug 10 14:40:53 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372853, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
I20260810 14:40:53.674285 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:53.662385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:37909 (local address 127.15.84.62:36385)
0810 14:40:53.662525 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:53.662529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:53.663390 (+ 861us) server_negotiation.cc:407] Connection header received
0810 14:40:53.664702 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:53.664705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:53.664753 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:53.664824 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:53.666024 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:53.666546 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:53.667724 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:53.667950 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:53.668042 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:53.670722 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:53.670727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:53.670730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:53.670763 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:53.672207 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:53.672692 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:53.672695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:53.672697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:53.672788 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:53.673077 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:53.673080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:53.673082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:53.673340 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:53.673436 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:53.673978 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 14:40:53.674114 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:40:53.675151 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:53.662772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.2:37909)
0810 14:40:53.663242 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:53.663285 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:53.664509 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:53.664979 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:53.664986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:53.665417 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:53.665909 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:53.665924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:53.666634 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:53.666636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:53.667606 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:53.667615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:53.668092 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:53.668097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:53.668274 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:53.668822 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:53.668839 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:53.670521 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:53.672316 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:53.672321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:53.672334 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:53.672592 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:53.672869 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:53.672871 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:53.672872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:53.673001 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:53.673430 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:53.673436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:53.673705 (+ 269us) client_negotiation.cc:769] Sending connection context
0810 14:40:53.673899 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 14:40:53.674189 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
I20260810 14:40:53.676371 25481 heartbeater.cc:347] Connected to a master server at 127.15.84.62:36385
I20260810 14:40:53.676623 25481 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:53.677135 25481 heartbeater.cc:511] Master 127.15.84.62:36385 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.15.84.3@KRBTEST.COM; defaulting to no policy
I20260810 14:40:53.678295 24976 ts_manager.cc:194] Registered new tserver with Master: f2561cc3377241c0a83657629ac58389 (127.15.84.2:35725)
I20260810 14:40:53.678929 24976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:37909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.3.keytab.
Entry for principal oryx/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.3.keytab.
I20260810 14:40:53.695796 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:36385
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.3.keytab
--principal=oryx/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:53.815569 25486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:53.815872 25486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:53.815917 25486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:53.820057 25486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:53.820132 25486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:53.820192 25486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:40:53.825730 25486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.3.keytab
--principal=oryx/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:53.826839 25486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:53.827870 25486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:53.838318 25499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:53.838332 25504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:53.838318 25498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:53.838732 25486 server_base.cc:1034] running on GCE node
I20260810 14:40:53.839178 25486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:53.840044 25486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:53.841213 25486 hybrid_clock.cc:648] HybridClock initialized: now 1786372853841177 us; error 60 us; skew 500 ppm
Aug 10 14:40:53 dist-test-slave-rmhg krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372853, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:53.844051 25486 init.cc:377] Logged in from keytab as oryx/127.15.84.3@KRBTEST.COM (short username oryx)
I20260810 14:40:53.845049 25486 webserver.cc:460] Webserver started at http://127.15.84.3:45575/ using document root <none> and password file <none>
I20260810 14:40:53.845687 25486 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:53.845759 25486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:53.845960 25486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:53.848078 25486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "9327ffe3276c48a49fa54211a6e32fff"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "89230b6c35b63ba6d0fb3fad5cf4aa03"
server_key_iv: "0850bca9e040c901331375a60e9e3ce5"
server_key_version: "encryptionkey@0"
I20260810 14:40:53.848654 25486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "9327ffe3276c48a49fa54211a6e32fff"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "89230b6c35b63ba6d0fb3fad5cf4aa03"
server_key_iv: "0850bca9e040c901331375a60e9e3ce5"
server_key_version: "encryptionkey@0"
I20260810 14:40:53.852787 25486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 14:40:53.855485 25512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:53.856848 25486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:40:53.856971 25486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "9327ffe3276c48a49fa54211a6e32fff"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "89230b6c35b63ba6d0fb3fad5cf4aa03"
server_key_iv: "0850bca9e040c901331375a60e9e3ce5"
server_key_version: "encryptionkey@0"
I20260810 14:40:53.857077 25486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:53.876570 25486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:53.879894 25486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:53.880097 25486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:53.880738 25486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:53.881742 25486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:53.881805 25486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:53.881861 25486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:53.881886 25486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:53.890969 25486 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:46843
I20260810 14:40:53.890990 25691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:46843 every 8 connection(s)
I20260810 14:40:53.891917 25486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:40:53.893445 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 25486
I20260810 14:40:53.893555 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:40:53.893887 15696 external_mini_cluster.cc:1428] Setting key a30921461f9c118cfad1158776de8029
Aug 10 14:40:53 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372853, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
I20260810 14:40:53.905990 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:53.893846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:43983 (local address 127.15.84.62:36385)
0810 14:40:53.894007 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:53.894011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:53.894939 (+ 928us) server_negotiation.cc:407] Connection header received
0810 14:40:53.896480 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:53.896482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:53.896529 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:53.896596 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:53.897817 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:53.898347 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:53.899504 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:53.899752 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:53.899834 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:53.902732 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:53.902736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:53.902738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:53.902768 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:53.904101 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:53.904577 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:53.904580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:53.904581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:53.904672 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:53.904944 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:53.904946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:53.904947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:53.905139 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:53.905250 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:53.905742 (+ 492us) server_negotiation.cc:299] Negotiation successful
0810 14:40:53.905855 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:40:53.907004 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:53.894268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.3:43983)
0810 14:40:53.894791 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:53.894855 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:53.896290 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:53.896744 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:53.896751 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:53.897117 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:53.897688 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:53.897704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:53.898442 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:53.898444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:53.899390 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:53.899400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:53.899936 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:53.899941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:53.900102 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:53.900700 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:53.900718 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:53.902519 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:53.904205 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:53.904210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:53.904224 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:53.904505 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:53.904737 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:53.904739 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:53.904740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:53.904883 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:53.905251 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:53.905256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:53.905525 (+ 269us) client_negotiation.cc:769] Sending connection context
0810 14:40:53.905706 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 14:40:53.905991 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":118,"threads_started":1}
I20260810 14:40:53.908085 25692 heartbeater.cc:347] Connected to a master server at 127.15.84.62:36385
I20260810 14:40:53.908331 25692 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:53.908835 25692 heartbeater.cc:511] Master 127.15.84.62:36385 requested a full tablet report, sending...
I20260810 14:40:53.909901 24976 ts_manager.cc:194] Registered new tserver with Master: 9327ffe3276c48a49fa54211a6e32fff (127.15.84.3:46843)
I20260810 14:40:53.910468 24976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:43983
I20260810 14:40:53.919883 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.84.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.84.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.84.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.61.keytab.
Entry for principal oryx/127.15.84.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.61.keytab.
I20260810 14:40:53.946478 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.61:36739
--webserver_interface=127.15.84.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.61.keytab
--principal=oryx/127.15.84.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:36385,127.15.84.61:36739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:54.063571 25701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:54.063885 25701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:54.063928 25701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:54.068097 25701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:54.068182 25701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:54.068202 25701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:54.068216 25701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:54.068230 25701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:54.073984 25701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:36385,127.15.84.61:36739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.61.keytab
--principal=oryx/127.15.84.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.61:36739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.84.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
W20260810 14:40:54.074378 25701 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.15.84.62:36385,127.15.84.61:36739'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 14:40:54.075551 25701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:54.076937 25701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:54.087018 25708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:54.087010 25710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:54.087018 25712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:54.087391 25701 server_base.cc:1034] running on GCE node
I20260810 14:40:54.088001 25701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:54.089236 25701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:54.090405 25701 hybrid_clock.cc:648] HybridClock initialized: now 1786372854090383 us; error 39 us; skew 500 ppm
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:54.093269 25701 init.cc:377] Logged in from keytab as oryx/127.15.84.61@KRBTEST.COM (short username oryx)
I20260810 14:40:54.094266 25701 webserver.cc:460] Webserver started at http://127.15.84.61:38701/ using document root <none> and password file <none>
I20260810 14:40:54.094844 25701 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:54.094915 25701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:54.095126 25701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:54.097361 25701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/data/instance:
uuid: "ee49a0d980e24a6a9d712deafedd3091"
format_stamp: "Formatted at 2026-08-10 14:40:54 on dist-test-slave-rmhg"
server_key: "fd4f67efc947109a244b183f761512f9"
server_key_iv: "1a370788418c1decacb68cfad8055007"
server_key_version: "encryptionkey@0"
I20260810 14:40:54.097971 25701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/wal/instance:
uuid: "ee49a0d980e24a6a9d712deafedd3091"
format_stamp: "Formatted at 2026-08-10 14:40:54 on dist-test-slave-rmhg"
server_key: "fd4f67efc947109a244b183f761512f9"
server_key_iv: "1a370788418c1decacb68cfad8055007"
server_key_version: "encryptionkey@0"
I20260810 14:40:54.102088 25701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:40:54.104801 25728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:54.106206 25701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:54.106346 25701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/wal
uuid: "ee49a0d980e24a6a9d712deafedd3091"
format_stamp: "Formatted at 2026-08-10 14:40:54 on dist-test-slave-rmhg"
server_key: "fd4f67efc947109a244b183f761512f9"
server_key_iv: "1a370788418c1decacb68cfad8055007"
server_key_version: "encryptionkey@0"
I20260810 14:40:54.106457 25701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:54.131842 25701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:54.135252 25701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:54.135459 25701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:54.142714 25701 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.61:36739
I20260810 14:40:54.143028 25811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.61:36739 every 8 connection(s)
I20260810 14:40:54.143810 25701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/data/info.pb
I20260810 14:40:54.144261 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 25701
I20260810 14:40:54.147372 25812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:54.152243 25812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.84.61@KRBTEST.COM
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.61@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
I20260810 14:40:54.159950 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.145911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.15.84.61:36739)
0810 14:40:54.146352 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.146371 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.146416 (+ 45us) server_negotiation.cc:407] Connection header received
0810 14:40:54.147273 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.147299 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.147737 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.148114 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.148780 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.150359 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.151324 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.152019 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.152194 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.154930 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.154938 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.154960 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.154989 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.156752 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.157220 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.157226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.157233 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.157384 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.157635 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.157638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.157639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.158151 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.158378 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.158667 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.158983 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":126,"threads_started":1}
I20260810 14:40:54.163695 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.154185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43904 (local address 127.15.84.62:36385)
0810 14:40:54.154344 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.154349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.154779 (+ 430us) server_negotiation.cc:407] Connection header received
0810 14:40:54.154921 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.154923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.154975 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.155043 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.155857 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.156418 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.157604 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.157849 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.157931 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.160321 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.160325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.160328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.160361 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.161982 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.162499 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.162502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.162504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.162594 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.162857 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.162860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.162861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.163093 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.163188 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.163474 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.163574 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:40:54.163749 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.154127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.0.0.1:43904)
0810 14:40:54.154613 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.154684 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.154829 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.155103 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.155110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.155530 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.155710 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.155719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.156540 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.156544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.157495 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.157504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.157964 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.157968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.158079 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.158631 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.158654 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.160153 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.162083 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.162096 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.162101 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.162396 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.162703 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.162705 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.162706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.162783 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.163200 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.163205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.163313 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.163429 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.163579 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":164,"threads_started":1}
I20260810 14:40:54.164860 25812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.61@KRBTEST.COM for oryx/127.15.84.61@KRBTEST.COM
I20260810 14:40:54.172334 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.165395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.15.84.61:36739)
0810 14:40:54.165502 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.165505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.165519 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:40:54.165613 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.165615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.165649 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.165726 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.166036 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.166883 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.167547 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.167772 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.167839 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.169646 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.169649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.169651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.169677 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.170893 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.171260 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.171263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.171265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.171331 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.171554 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.171558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.171559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.171739 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.171820 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.172071 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.172195 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":38}
I20260810 14:40:54.172334 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.165352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:36739 (local address 127.0.0.1:57580)
0810 14:40:54.165472 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.165486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.165551 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.165743 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.165744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.165868 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.165960 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.165965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.166962 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.166964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.167473 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.167479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.167873 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.167875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.167947 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.168328 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.168338 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.169527 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.170991 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.170994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.170996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.171179 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.171404 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.171407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.171409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.171483 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.171821 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.171824 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.171907 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.171993 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.172140 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 14:40:54.175446 25812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: Bootstrap starting.
I20260810 14:40:54.177925 25812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:54.178632 25812 log.cc:824] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:54.180542 25812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: No bootstrap required, opened a new log
I20260810 14:40:54.183594 25812 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } }
I20260810 14:40:54.183830 25812 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:54.183873 25812 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee49a0d980e24a6a9d712deafedd3091, State: Initialized, Role: FOLLOWER
I20260810 14:40:54.184265 25812 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } }
I20260810 14:40:54.184737 25823 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } } }
I20260810 14:40:54.184882 25823 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:54.185497 25812 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 14:40:54.190325 25838 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:40:54.190423 25838 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:40:54.192317 25701 master_runner.cc:428] Detected that this master ee49a0d980e24a6a9d712deafedd3091 is joining an existing cluster
I20260810 14:40:54.192387 25701 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.84.61:36739
I20260810 14:40:54.204099 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.195388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.15.84.61:36739)
0810 14:40:54.195506 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.195510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.196056 (+ 546us) server_negotiation.cc:407] Connection header received
0810 14:40:54.196193 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.196196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.196232 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.196293 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.196796 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.197593 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.198846 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.199178 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.199246 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.200806 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.200810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.200813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.200839 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.202522 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.202958 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.202961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.202963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.203047 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.203290 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.203293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.203295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.203469 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.203551 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.203778 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.203929 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":50}
I20260810 14:40:54.204113 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.195315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:36739 (local address 127.0.0.1:57586)
0810 14:40:54.195990 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.196002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.196102 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.196322 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.196325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.196543 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.196685 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.196693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.197725 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.197729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.198725 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.198738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.199382 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.199385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.199475 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.199953 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.199968 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.200433 (+ 465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.202610 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.202613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.202615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.202870 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.203128 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.203130 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.203132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.203207 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.203551 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.203554 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.203640 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.203724 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.203883 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":33,"spinlock_wait_cycles":1920,"thread_start_us":88,"threads_started":1}
I20260810 14:40:54.204276 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.194960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.15.84.62:36385)
0810 14:40:54.195112 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.195116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.195710 (+ 594us) server_negotiation.cc:407] Connection header received
0810 14:40:54.195861 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.195865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.195919 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.196005 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.196694 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.197351 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.198261 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.198483 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.198574 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.200277 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.200281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.200284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.200312 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.202130 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.202766 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.202770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.202772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.202863 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.203340 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.203344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.203346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.203573 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.203643 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.203982 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.204107 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:40:54.204825 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.194859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.0.0.1:43918)
0810 14:40:54.195618 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.195629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.195759 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.196055 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.196059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.196330 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.196539 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.196549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.197454 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.197458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.198145 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.198155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.198601 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.198604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.198758 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.199659 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.199678 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.200147 (+ 469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.202286 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.202290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.202292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.202659 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.203155 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.203158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.203160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.203236 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.203715 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.203717 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.203811 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.204497 (+ 686us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.204644 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":105,"spinlock_wait_cycles":3328,"thread_start_us":118,"threads_started":1}
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372853, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.62@KRBTEST.COM for oryx/127.15.84.61@KRBTEST.COM
I20260810 14:40:54.218796 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.209516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.15.84.61:36739)
0810 14:40:54.209622 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.209626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.210011 (+ 385us) server_negotiation.cc:407] Connection header received
0810 14:40:54.210217 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.210220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.210270 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.210335 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.211159 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.211655 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.212514 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.212738 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.212823 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.215193 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.215196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.215198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.215220 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.216767 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.217392 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.217396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.217398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.217495 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.217773 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.217776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.217778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.217998 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.218091 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.218503 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.218630 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 14:40:54.218837 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.209507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:36739 (local address 127.0.0.1:57600)
0810 14:40:54.209888 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.209935 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.210071 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.210528 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.210533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.210930 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.211047 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.211053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.211756 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.211759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.212405 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.212414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.212838 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.212842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.212954 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.213533 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.213547 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.215060 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.216886 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.216898 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.216907 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.217290 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.217584 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.217588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.217590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.217684 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.218102 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.218109 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.218265 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.218432 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.218655 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":120,"threads_started":1}
I20260810 14:40:54.220849 24976 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.84.61:36739: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091"
member_type: NON_VOTER
last_known_addr {
host: "127.15.84.61"
port: 36739
}
attrs {
promote: true
}
}
dest_uuid: "2767b706cceb48a5bf5440ff912fdea8"
cas_config_opid_index: -1
I20260810 14:40:54.221704 24976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.15.84.60@KRBTEST.COM; defaulting to no policy
I20260810 14:40:54.224313 25860 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ee49a0d980e24a6a9d712deafedd3091 (127.15.84.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } }
I20260810 14:40:54.225126 25772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:54.226279 25860 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.84.61:36739
I20260810 14:40:54.226620 25860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } }
I20260810 14:40:54.226837 25860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:54.226938 25701 master.cc:537] Master@127.15.84.61:36739 shutting down...
I20260810 14:40:54.227209 25772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 FOLLOWER]: Refusing update from remote peer 2767b706cceb48a5bf5440ff912fdea8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 14:40:54.228042 25859 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.84.60@KRBTEST.COM" created.
W20260810 14:40:54.231113 24956 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 14:40:54.232218 24956 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 -> Peer ee49a0d980e24a6a9d712deafedd3091 (127.15.84.61:36739): Couldn't send request to peer ee49a0d980e24a6a9d712deafedd3091. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 14:40:54.233261 25701 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:54.233585 25701 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:54.233695 25701 tablet_replica.cc:330] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: stopping tablet replica
I20260810 14:40:54.238785 25701 master.cc:552] Master@127.15.84.61:36739 shutdown complete.
I20260810 14:40:54.238921 25701 master_runner.cc:305] Clearing existing system tablet
I20260810 14:40:54.239256 25701 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:54.239403 25701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:54.240942 25701 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:54.241989 25868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:54.242208 25701 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:54.242339 25701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 14:40:54.242396 25701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/wal
uuid: "ee49a0d980e24a6a9d712deafedd3091"
format_stamp: "Formatted at 2026-08-10 14:40:54 on dist-test-slave-rmhg"
server_key: "fd4f67efc947109a244b183f761512f9"
server_key_iv: "1a370788418c1decacb68cfad8055007"
server_key_version: "encryptionkey@0"
I20260810 14:40:54.244963 25701 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:54.247737 25701 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 14:40:54.247850 25701 log.cc:1196] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 14:40:54.248396 25701 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: Deleting consensus metadata
I20260810 14:40:54.248557 25701 master_runner.cc:315] Copying system tablet from 127.15.84.62:36385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.84.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.60.keytab.
Entry for principal oryx/127.15.84.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.60.keytab.
I20260810 14:40:54.249927 25701 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: tablet copy: Beginning tablet copy session from remote peer at address 127.15.84.62:36385
I20260810 14:40:54.250413 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.60:46403
--webserver_interface=127.15.84.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.60.keytab
--principal=oryx/127.15.84.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:36385,127.15.84.61:36739,127.15.84.60:46403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:40:54.261889 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.252786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.15.84.62:36385)
0810 14:40:54.252924 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.252928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.253246 (+ 318us) server_negotiation.cc:407] Connection header received
0810 14:40:54.253352 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.253355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.253416 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.253503 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.254071 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.254929 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.256146 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.256503 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.256616 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.258023 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.258027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.258030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.258061 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.259723 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.260237 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.260241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.260243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.260347 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.260645 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.260648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.260650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.260980 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.261082 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.261548 (+ 466us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.261708 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:40:54.262900 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.252666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.0.0.1:43934)
0810 14:40:54.253072 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.253087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.253257 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.253540 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.253543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.253789 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.253941 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.253949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.255067 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.255072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.256006 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.256021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.256636 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.256640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.256752 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.257372 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.257388 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.257852 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.259845 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.259850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.259852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.260124 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.260445 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.260448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.260451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.260552 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.261086 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.261090 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.261313 (+ 223us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.262483 (+ 1170us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.262686 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":108,"threads_started":1}
I20260810 14:40:54.264818 25009 tablet_copy_service.cc:140] P 2767b706cceb48a5bf5440ff912fdea8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ee49a0d980e24a6a9d712deafedd3091 ({username='oryx', principal='oryx/127.15.84.61@KRBTEST.COM'} at 127.0.0.1:43934)
I20260810 14:40:54.265040 25009 tablet_copy_service.cc:161] P 2767b706cceb48a5bf5440ff912fdea8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ee49a0d980e24a6a9d712deafedd3091 at {username='oryx', principal='oryx/127.15.84.61@KRBTEST.COM'} at 127.0.0.1:43934: session id = ee49a0d980e24a6a9d712deafedd3091-00000000000000000000000000000000
I20260810 14:40:54.270007 25009 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 14:40:54.272061 25701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 14:40:54.274610 25701 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: tablet copy: Starting download of 0 data blocks...
I20260810 14:40:54.274921 25701 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: tablet copy: Starting download of 1 WAL segments...
I20260810 14:40:54.276974 25701 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 14:40:54.278998 25009 tablet_copy_service.cc:342] P 2767b706cceb48a5bf5440ff912fdea8: Request end of tablet copy session ee49a0d980e24a6a9d712deafedd3091-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.84.61@KRBTEST.COM'} at 127.0.0.1:43934
I20260810 14:40:54.279155 25009 tablet_copy_service.cc:433] P 2767b706cceb48a5bf5440ff912fdea8: ending tablet copy session ee49a0d980e24a6a9d712deafedd3091-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ee49a0d980e24a6a9d712deafedd3091
I20260810 14:40:54.287235 25701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:54.288959 25881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:54.290117 25882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:54.290405 25701 server_base.cc:1034] running on GCE node
W20260810 14:40:54.290421 25885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:54.290652 25701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:54.290963 25701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:54.292097 25701 hybrid_clock.cc:648] HybridClock initialized: now 1786372854292086 us; error 31 us; skew 500 ppm
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:54.293920 25701 init.cc:377] Logged in from keytab as oryx/127.15.84.61@KRBTEST.COM (short username oryx)
I20260810 14:40:54.294459 25701 webserver.cc:460] Webserver started at http://127.15.84.61:43469/ using document root <none> and password file <none>
I20260810 14:40:54.294711 25701 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:54.294807 25701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:54.296268 25701 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:54.297326 25897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:54.297588 25701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:54.297662 25701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/wal
uuid: "ee49a0d980e24a6a9d712deafedd3091"
format_stamp: "Formatted at 2026-08-10 14:40:54 on dist-test-slave-rmhg"
server_key: "fd4f67efc947109a244b183f761512f9"
server_key_iv: "1a370788418c1decacb68cfad8055007"
server_key_version: "encryptionkey@0"
I20260810 14:40:54.297745 25701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:54.313257 25701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:54.315557 25701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:54.320973 25701 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.61:36739
I20260810 14:40:54.321029 25973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.61:36739 every 8 connection(s)
I20260810 14:40:54.321684 25701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-1/data/info.pb
I20260810 14:40:54.323760 25974 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:40:54.324962 25974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: Bootstrap starting.
I20260810 14:40:54.335759 25974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:40:54.336087 25974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: Bootstrap complete.
I20260810 14:40:54.336743 25974 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } }
I20260810 14:40:54.336889 25974 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee49a0d980e24a6a9d712deafedd3091, State: Initialized, Role: LEARNER
I20260810 14:40:54.337002 25974 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } }
I20260810 14:40:54.337387 25979 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } }
I20260810 14:40:54.337504 25974 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:54.337592 25979 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:40:54.340554 25993 catalog_manager.cc:1259] Loaded cluster ID: 53b69ac8f06e4caaaecacf0a13e1c527
I20260810 14:40:54.340718 25993 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: loading cluster ID for follower catalog manager: success
I20260810 14:40:54.343034 25993 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: acquiring CA information for follower catalog manager: success
I20260810 14:40:54.344633 25993 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 14:40:54.393057 25878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:54.393427 25878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:54.393477 25878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:54.397704 25878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:54.397790 25878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:54.397818 25878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:54.397842 25878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:54.397873 25878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:54.403663 25878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:36385,127.15.84.61:36739,127.15.84.60:46403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.60.keytab
--principal=oryx/127.15.84.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.60:46403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.84.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:54.404901 25878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:54.405972 25878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:54.414788 25998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:54.414804 26001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:54.414846 25878 server_base.cc:1034] running on GCE node
W20260810 14:40:54.414798 26005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:54.415709 25878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:54.416847 25878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:54.418020 25878 hybrid_clock.cc:648] HybridClock initialized: now 1786372854417997 us; error 35 us; skew 500 ppm
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:54.420840 25878 init.cc:377] Logged in from keytab as oryx/127.15.84.60@KRBTEST.COM (short username oryx)
I20260810 14:40:54.421842 25878 webserver.cc:460] Webserver started at http://127.15.84.60:35983/ using document root <none> and password file <none>
I20260810 14:40:54.422401 25878 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:54.422461 25878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:54.422647 25878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:54.424855 25878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/data/instance:
uuid: "3b9146920cbc4e7a9dd800bb40b11d7d"
format_stamp: "Formatted at 2026-08-10 14:40:54 on dist-test-slave-rmhg"
server_key: "37d0730e447c8141ee656a9a2245dd41"
server_key_iv: "9fe6c57770e9b8e0c37e554d4f3a20ed"
server_key_version: "encryptionkey@0"
I20260810 14:40:54.425447 25878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/wal/instance:
uuid: "3b9146920cbc4e7a9dd800bb40b11d7d"
format_stamp: "Formatted at 2026-08-10 14:40:54 on dist-test-slave-rmhg"
server_key: "37d0730e447c8141ee656a9a2245dd41"
server_key_iv: "9fe6c57770e9b8e0c37e554d4f3a20ed"
server_key_version: "encryptionkey@0"
I20260810 14:40:54.429467 25878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:40:54.432375 26017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:54.433830 25878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:54.433967 25878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/wal
uuid: "3b9146920cbc4e7a9dd800bb40b11d7d"
format_stamp: "Formatted at 2026-08-10 14:40:54 on dist-test-slave-rmhg"
server_key: "37d0730e447c8141ee656a9a2245dd41"
server_key_iv: "9fe6c57770e9b8e0c37e554d4f3a20ed"
server_key_version: "encryptionkey@0"
I20260810 14:40:54.434092 25878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:54.450692 25878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:54.454120 25878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:54.454339 25878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:54.457872 25270 heartbeater.cc:503] Master 127.15.84.62:36385 was elected leader, sending a full tablet report...
I20260810 14:40:54.462301 25878 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.60:46403
I20260810 14:40:54.462587 26094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.60:46403 every 8 connection(s)
I20260810 14:40:54.463402 25878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/data/info.pb
I20260810 14:40:54.465586 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 25878
I20260810 14:40:54.466805 26095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:54.471760 26095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.84.60@KRBTEST.COM
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.60@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
I20260810 14:40:54.482539 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.467244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52544 (local address 127.15.84.60:46403)
0810 14:40:54.467710 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.467731 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.467776 (+ 45us) server_negotiation.cc:407] Connection header received
0810 14:40:54.468668 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.468694 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.469172 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.469593 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.470300 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.471846 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.472760 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.473504 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.473641 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.476565 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.476571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.476581 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.476612 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.478708 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.479261 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.479267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.479273 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.479403 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.479726 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.479729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.479731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.480217 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.480444 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.481023 (+ 579us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.481421 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":130,"threads_started":1}
I20260810 14:40:54.483832 26102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.473900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.0.0.1:43940)
0810 14:40:54.474373 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.474419 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.474547 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.474839 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.474846 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.475232 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.475392 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.475401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.476326 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.476329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.477145 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.477184 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.477716 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.477720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.477853 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.478275 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.478291 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.480111 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.482065 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.482074 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.482079 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.482318 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.482612 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.482614 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.482616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.482710 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.483184 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.483191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.483307 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.483432 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.483635 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":174,"threads_started":1}
I20260810 14:40:54.484017 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.474033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.15.84.62:36385)
0810 14:40:54.474206 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.474211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.474503 (+ 292us) server_negotiation.cc:407] Connection header received
0810 14:40:54.474649 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.474652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.474704 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.474770 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.475531 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.476176 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.477290 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.477582 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.477697 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.480287 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.480292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.480295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.480328 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.481931 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.482418 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.482422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.482424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.482521 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.482857 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.482861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.482862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.483067 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.483161 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.483505 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.483665 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:40:54.485090 26095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.60@KRBTEST.COM for oryx/127.15.84.61@KRBTEST.COM
I20260810 14:40:54.493598 26102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.485600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:36739 (local address 127.0.0.1:57614)
0810 14:40:54.485722 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.485736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.485807 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.486237 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.486240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.486403 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.486558 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.486565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.487502 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.487505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.488096 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.488105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.488565 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.488569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.488663 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.489124 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.489147 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.490476 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.492045 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.492049 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.492052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.492256 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.492536 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.492539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.492541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.492624 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.493019 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.493023 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.493122 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.493258 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.493413 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 14:40:54.493599 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.485752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.15.84.61:36739)
0810 14:40:54.486032 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.486035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.486047 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:40:54.486083 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.486086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.486124 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.486204 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.486717 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.487394 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.488221 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.488464 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.488555 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.490643 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.490647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.490649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.490677 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.491919 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.492353 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.492357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.492359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.492447 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.492712 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.492716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.492718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.492926 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.493018 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.493319 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.493445 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":112,"threads_started":1}
I20260810 14:40:54.494485 26095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.60@KRBTEST.COM for oryx/127.15.84.60@KRBTEST.COM
I20260810 14:40:54.502828 26102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.495004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:46403 (local address 127.0.0.1:52558)
0810 14:40:54.495148 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.495162 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.495235 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.495412 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.495414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.495560 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.495644 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.495649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.496335 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.496338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.496864 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.496872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.497872 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.497876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.497988 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.498538 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.498555 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.499798 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.501419 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.501422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.501424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.501600 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.501919 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.501922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.501924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.502001 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.502331 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.502333 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.502418 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.502500 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.502643 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 14:40:54.502830 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.495121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.15.84.60:46403)
0810 14:40:54.495218 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.495221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.495233 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:40:54.495296 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.495298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.495332 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.495393 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.495727 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.496234 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.496949 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.497755 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.497839 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.499936 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.499939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.499941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.499963 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.501277 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.501689 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.501693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.501695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.501835 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.502074 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.502077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.502079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.502248 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.502331 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.502554 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.502711 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":34}
I20260810 14:40:54.505873 26095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: Bootstrap starting.
I20260810 14:40:54.508285 26095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:54.509063 26095 log.cc:824] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:54.511109 26095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: No bootstrap required, opened a new log
I20260810 14:40:54.514472 26095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } }
I20260810 14:40:54.514716 26095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:54.514755 26095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9146920cbc4e7a9dd800bb40b11d7d, State: Initialized, Role: FOLLOWER
I20260810 14:40:54.515283 26095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } }
I20260810 14:40:54.515858 26108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } } }
I20260810 14:40:54.515990 26108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:54.516700 26095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: configured and running, proceeding with master startup.
W20260810 14:40:54.521800 26119 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:40:54.521889 26119 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:40:54.525556 25878 master_runner.cc:428] Detected that this master 3b9146920cbc4e7a9dd800bb40b11d7d is joining an existing cluster
I20260810 14:40:54.525646 25878 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.84.60:46403
I20260810 14:40:54.536473 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 25069
I20260810 14:40:54.553994 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:33165
--local_ip_for_outbound_sockets=127.15.84.1
--tserver_master_addrs=127.15.84.62:36385
--webserver_port=40147
--webserver_interface=127.15.84.1
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.1.keytab
--principal=oryx/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.84.62:36385,127.15.84.61:36739
--tserver_master_addrs=127.15.84.62:36385,127.15.84.61:36739,127.15.84.60:46403 with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:40:54.555014 26108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:54.555271 26108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } }
I20260810 14:40:54.556476 26108 leader_election.cc:288] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2767b706cceb48a5bf5440ff912fdea8 (127.15.84.62:36385), ee49a0d980e24a6a9d712deafedd3091 (127.15.84.61:36739)
I20260810 14:40:54.557066 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2767b706cceb48a5bf5440ff912fdea8" is_pre_election: true
I20260810 14:40:54.557114 25935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee49a0d980e24a6a9d712deafedd3091" is_pre_election: true
I20260810 14:40:54.557374 25935 raft_consensus.cc:1799] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 LEARNER]: Handling vote request from an unknown peer 3b9146920cbc4e7a9dd800bb40b11d7d
I20260810 14:40:54.557518 25935 raft_consensus.cc:2396] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 3b9146920cbc4e7a9dd800bb40b11d7d for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260810 14:40:54.558290 26019 leader_election.cc:302] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b9146920cbc4e7a9dd800bb40b11d7d; no voters: 2767b706cceb48a5bf5440ff912fdea8, ee49a0d980e24a6a9d712deafedd3091
I20260810 14:40:54.559042 26108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:54.560868 26108 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 14:40:54.563097 26137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.528630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:46403 (local address 127.0.0.1:52562)
0810 14:40:54.529232 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.529246 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.529366 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.529688 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.529691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.529927 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.530082 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.530090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.530997 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.530999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.531883 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.531894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.532661 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.532665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.532756 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.533352 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.533365 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.557723 (+ 24358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.561352 (+ 3629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.561356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.561358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.561605 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.561890 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.561893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.561895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.561974 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.562504 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.562507 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.562620 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.562725 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.562885 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":184,"spinlock_wait_cycles":4352,"thread_start_us":110,"threads_started":1}
I20260810 14:40:54.563725 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.528415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.15.84.60:46403)
0810 14:40:54.528539 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.528543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.529311 (+ 768us) server_negotiation.cc:407] Connection header received
0810 14:40:54.529454 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.529457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.529493 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.529541 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.530189 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.530876 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.532011 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.532424 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.532519 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.558075 (+ 25556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.558079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.558082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.558116 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.561221 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.561714 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.561718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.561719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.561800 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.562061 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.562064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.562066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.562372 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.562468 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.563317 (+ 849us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.563581 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":41}
I20260810 14:40:54.564661 26133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.528396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.0.0.1:43950)
0810 14:40:54.528750 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.528767 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.528931 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.529387 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.529390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.529633 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.529778 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.529787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.530919 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.530922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.532379 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.532390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.533281 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.533284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.533525 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.559028 (+ 25503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.559044 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.559481 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.562568 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.562572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.562574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.562815 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.563135 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.563139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.563141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.563234 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.564128 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.564131 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.564250 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.564348 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.564508 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":201,"thread_start_us":127,"threads_started":1}
I20260810 14:40:54.565457 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.528486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.15.84.62:36385)
0810 14:40:54.528759 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.528764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.528846 (+ 82us) server_negotiation.cc:407] Connection header received
0810 14:40:54.529045 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.529049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.529104 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.529213 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.529924 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.530775 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.532531 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.532892 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.533322 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.560153 (+ 26831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.560158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.560162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.560200 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.562400 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.562930 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.562934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.562936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.563028 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.563575 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.563579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.563581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.563805 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.563885 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.565141 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.565308 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260810 14:40:54.570783 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.529232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57622 (local address 127.15.84.61:36739)
0810 14:40:54.529363 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.529367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.529640 (+ 273us) server_negotiation.cc:407] Connection header received
0810 14:40:54.529801 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.529804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.529854 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.529905 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.530565 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.531322 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.532589 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.532938 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.533012 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.566003 (+ 32991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.566007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.566010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.566045 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.568573 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.569293 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.569297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.569299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.569393 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.569829 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.569832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.569834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.570069 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.570149 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.570491 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.570632 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:40:54.571931 26134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.528884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:36739 (local address 127.0.0.1:57622)
0810 14:40:54.529552 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.529566 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.529697 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.530045 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.530048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.530250 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.530416 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.530423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.531458 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.531461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.532455 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.532463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.533176 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.533180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.533272 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.533709 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.533720 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.565792 (+ 32072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.568718 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.568721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.568724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.569144 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.569502 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.569505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.569507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.569730 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.570218 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.570221 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.570313 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.571584 (+ 1271us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.571751 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":50,"thread_start_us":69,"threads_started":1}
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372853, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.62@KRBTEST.COM for oryx/127.15.84.60@KRBTEST.COM
I20260810 14:40:54.580019 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.570610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.15.84.60:46403)
0810 14:40:54.570717 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.570722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.570733 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:40:54.570801 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.570804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.570852 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.570922 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.571503 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.572297 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.573469 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.573811 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.573917 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.576388 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.576392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.576395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.576424 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.578181 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.578713 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.578716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.578718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.578806 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.579094 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.579097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.579098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.579342 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.579440 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.579727 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.579856 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 14:40:54.580020 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.570373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:46403 (local address 127.0.0.1:52578)
0810 14:40:54.570486 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.570501 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.570581 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.571048 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.571050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.571265 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.571374 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.571381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.572421 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.572425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.573351 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.573363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.573923 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.573927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.574030 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.574526 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.574540 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.576130 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.578322 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.578326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.578329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.578603 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.578891 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.578902 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.578904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.579003 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.579437 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.579440 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.579562 (+ 122us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.579672 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.579844 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 14:40:54.582334 24976 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.84.60:46403: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d"
member_type: NON_VOTER
last_known_addr {
host: "127.15.84.60"
port: 46403
}
attrs {
promote: true
}
}
dest_uuid: "2767b706cceb48a5bf5440ff912fdea8"
cas_config_opid_index: 5
I20260810 14:40:54.583029 24976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } }
I20260810 14:40:54.584223 25859 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3b9146920cbc4e7a9dd800bb40b11d7d (127.15.84.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } } }
I20260810 14:40:54.585093 26059 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 FOLLOWER]: Refusing update from remote peer 2767b706cceb48a5bf5440ff912fdea8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 14:40:54.585875 25859 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.84.60:46403
I20260810 14:40:54.586051 25859 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:54.586760 25859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } } }
I20260810 14:40:54.586861 25859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:54.590261 26059 raft_consensus.cc:858] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } }
W20260810 14:40:54.590378 26059 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } }
I20260810 14:40:54.592172 26144 mvcc.cc:205] Tried to move back new op lower bound from 7316983206459748352 to 7316983206296436736. Current Snapshot: MvccSnapshot[applied={T|T < 7316983206335303680}]
I20260810 14:40:54.592519 25878 master.cc:537] Master@127.15.84.60:46403 shutting down...
I20260810 14:40:54.596681 26059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ee49a0d980e24a6a9d712deafedd3091 (127.15.84.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } }
I20260810 14:40:54.599030 26108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2767b706cceb48a5bf5440ff912fdea8. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } }
I20260810 14:40:54.599193 26108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 14:40:54.599280 26108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } }
I20260810 14:40:54.599344 26108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 14:40:54.599351 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.587163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:36739 (local address 127.0.0.1:57624)
0810 14:40:54.587992 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.588007 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.588092 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.594610 (+ 6518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.594614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.594647 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:54.594769 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.594778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.595917 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.595922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.597519 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.597558 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.598840 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.598844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.598946 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.599008 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.599079 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.599175 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747}
I20260810 14:40:54.599607 26151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } }
I20260810 14:40:54.599694 26151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 14:40:54.600242 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.587434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57624 (local address 127.15.84.61:36739)
0810 14:40:54.594206 (+ 6772us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.594211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.594231 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:40:54.594294 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.594297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.594368 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.594506 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:54.594931 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.595751 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.597738 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.598527 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.598626 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.599979 (+ 1353us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.600026 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.600108 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6654}
I20260810 14:40:54.600937 25935 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 LEARNER]: Refusing update from remote peer 2767b706cceb48a5bf5440ff912fdea8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 14:40:54.600931 25860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } } }
I20260810 14:40:54.601073 25860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:54.601770 25878 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260810 14:40:54.601940 26143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 14:40:54.602026 25878 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260810 14:40:54.602145 25878 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: stopping tablet replica
W20260810 14:40:54.602481 24954 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 -> Peer 3b9146920cbc4e7a9dd800bb40b11d7d (127.15.84.60:46403): Couldn't send request to peer 3b9146920cbc4e7a9dd800bb40b11d7d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 14:40:54.603885 25935 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3b9146920cbc4e7a9dd800bb40b11d7d (127.15.84.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } } }
I20260810 14:40:54.605825 26161 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } } }
I20260810 14:40:54.606015 26161 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:40:54.606244 25979 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2767b706cceb48a5bf5440ff912fdea8. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: NON_VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: true } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } } }
I20260810 14:40:54.606374 25979 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:40:54.607125 25860 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8: attempting to promote NON_VOTER ee49a0d980e24a6a9d712deafedd3091 to VOTER
I20260810 14:40:54.607582 25860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } }
W20260810 14:40:54.609654 24954 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 -> Peer 3b9146920cbc4e7a9dd800bb40b11d7d (127.15.84.60:46403): Couldn't send request to peer 3b9146920cbc4e7a9dd800bb40b11d7d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 14:40:54.610227 25932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 LEARNER]: Refusing update from remote peer 2767b706cceb48a5bf5440ff912fdea8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 14:40:54.611227 25859 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 14:40:54.613112 25859 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ee49a0d980e24a6a9d712deafedd3091 (127.15.84.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } } }
I20260810 14:40:54.614687 25859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } } }
I20260810 14:40:54.614809 25859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:54.615460 26143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } } }
I20260810 14:40:54.615571 26143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:54.616096 25932 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ee49a0d980e24a6a9d712deafedd3091 (127.15.84.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } } }
I20260810 14:40:54.618054 25979 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } } }
I20260810 14:40:54.618201 25979 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:54.626806 25878 master.cc:552] Master@127.15.84.60:46403 shutdown complete.
I20260810 14:40:54.626891 25878 master_runner.cc:305] Clearing existing system tablet
I20260810 14:40:54.627174 25878 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:54.627251 25878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:54.628671 25878 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:54.629675 26164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:54.629957 25878 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:54.630156 25878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 14:40:54.630214 25878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/wal
uuid: "3b9146920cbc4e7a9dd800bb40b11d7d"
format_stamp: "Formatted at 2026-08-10 14:40:54 on dist-test-slave-rmhg"
server_key: "37d0730e447c8141ee656a9a2245dd41"
server_key_iv: "9fe6c57770e9b8e0c37e554d4f3a20ed"
server_key_version: "encryptionkey@0"
I20260810 14:40:54.632282 25878 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:54.634950 25878 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 14:40:54.635102 25878 log.cc:1196] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 14:40:54.635453 25878 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: Deleting consensus metadata
I20260810 14:40:54.635664 25878 master_runner.cc:315] Copying system tablet from 127.15.84.62:36385
I20260810 14:40:54.636994 25878 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: tablet copy: Beginning tablet copy session from remote peer at address 127.15.84.62:36385
I20260810 14:40:54.647274 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.637795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.15.84.62:36385)
0810 14:40:54.637956 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.637961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.638257 (+ 296us) server_negotiation.cc:407] Connection header received
0810 14:40:54.638418 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.638422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.638475 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.638528 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.639313 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.640147 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.641449 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.641790 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.641874 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.643643 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.643647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.643650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.643682 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.645363 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.645879 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.645883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.645884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.645979 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.646273 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.646276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.646278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.646506 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.646578 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.646978 (+ 400us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.647106 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 14:40:54.648367 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.637772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.0.0.1:43956)
0810 14:40:54.638135 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.638151 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.638291 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.638747 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.638750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.639001 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.639161 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.639169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.640311 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.640314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.641281 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.641294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.642140 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.642144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.642246 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.642844 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.642859 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.643474 (+ 615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.645494 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.645498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.645500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.645763 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.646085 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.646088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.646089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.646177 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.646650 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.646653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.646791 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.647619 (+ 828us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.647795 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":97,"threads_started":1}
I20260810 14:40:54.648960 25009 tablet_copy_service.cc:140] P 2767b706cceb48a5bf5440ff912fdea8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3b9146920cbc4e7a9dd800bb40b11d7d ({username='oryx', principal='oryx/127.15.84.60@KRBTEST.COM'} at 127.0.0.1:43956)
I20260810 14:40:54.649061 25009 tablet_copy_service.cc:161] P 2767b706cceb48a5bf5440ff912fdea8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3b9146920cbc4e7a9dd800bb40b11d7d at {username='oryx', principal='oryx/127.15.84.60@KRBTEST.COM'} at 127.0.0.1:43956: session id = 3b9146920cbc4e7a9dd800bb40b11d7d-00000000000000000000000000000000
I20260810 14:40:54.650240 25009 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 14:40:54.651724 25878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 14:40:54.654309 25878 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: tablet copy: Starting download of 0 data blocks...
I20260810 14:40:54.654532 25878 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: tablet copy: Starting download of 1 WAL segments...
I20260810 14:40:54.656194 25878 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 14:40:54.658241 25009 tablet_copy_service.cc:342] P 2767b706cceb48a5bf5440ff912fdea8: Request end of tablet copy session 3b9146920cbc4e7a9dd800bb40b11d7d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.84.60@KRBTEST.COM'} at 127.0.0.1:43956
I20260810 14:40:54.658331 25009 tablet_copy_service.cc:433] P 2767b706cceb48a5bf5440ff912fdea8: ending tablet copy session 3b9146920cbc4e7a9dd800bb40b11d7d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3b9146920cbc4e7a9dd800bb40b11d7d
I20260810 14:40:54.667680 25878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:54.669457 26173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:54.669705 26177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:54.669817 26174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:54.673736 25878 server_base.cc:1034] running on GCE node
I20260810 14:40:54.673941 25878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:54.674306 25878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:54.675469 25878 hybrid_clock.cc:648] HybridClock initialized: now 1786372854675449 us; error 47 us; skew 500 ppm
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:54.677259 25878 init.cc:377] Logged in from keytab as oryx/127.15.84.60@KRBTEST.COM (short username oryx)
I20260810 14:40:54.677814 25878 webserver.cc:460] Webserver started at http://127.15.84.60:42341/ using document root <none> and password file <none>
I20260810 14:40:54.678030 25878 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:54.678081 25878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:54.679559 25878 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:54.680742 26186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:54.681084 25481 heartbeater.cc:503] Master 127.15.84.62:36385 was elected leader, sending a full tablet report...
I20260810 14:40:54.681883 25878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:40:54.681967 25878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/wal
uuid: "3b9146920cbc4e7a9dd800bb40b11d7d"
format_stamp: "Formatted at 2026-08-10 14:40:54 on dist-test-slave-rmhg"
server_key: "37d0730e447c8141ee656a9a2245dd41"
server_key_iv: "9fe6c57770e9b8e0c37e554d4f3a20ed"
server_key_version: "encryptionkey@0"
I20260810 14:40:54.682061 25878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:54.716553 25878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
W20260810 14:40:54.717713 26140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:54.718070 26140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:54.718151 26140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 14:40:54.718425 25878 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260810 14:40:54.722532 26140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:54.722597 26140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:54.722697 26140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:40:54.723031 25878 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.60:46403
I20260810 14:40:54.723814 25878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/master-2/data/info.pb
I20260810 14:40:54.724143 26281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.60:46403 every 8 connection(s)
I20260810 14:40:54.725548 26284 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:40:54.726522 26284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: Bootstrap starting.
I20260810 14:40:54.728617 26140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.1.keytab
--principal=oryx/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:33165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=40147
--tserver_master_addrs=127.15.84.62:36385,127.15.84.61:36739,127.15.84.60:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:54.730132 26140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:54.731221 26140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:40:54.733683 26284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:40:54.733933 26284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: Bootstrap complete.
I20260810 14:40:54.734441 26284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } }
I20260810 14:40:54.734558 26284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b9146920cbc4e7a9dd800bb40b11d7d, State: Initialized, Role: LEARNER
I20260810 14:40:54.734659 26284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } }
I20260810 14:40:54.734994 26289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } } }
I20260810 14:40:54.735118 26289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: This master's current role is: LEARNER
I20260810 14:40:54.735133 26284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:54.738850 26303 catalog_manager.cc:1259] Loaded cluster ID: 53b69ac8f06e4caaaecacf0a13e1c527
I20260810 14:40:54.738909 26303 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: loading cluster ID for follower catalog manager: success
I20260810 14:40:54.740542 26303 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: acquiring CA information for follower catalog manager: success
W20260810 14:40:54.742100 26308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:54.742331 26311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:54.742444 26303 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:40:54.742619 26140 server_base.cc:1034] running on GCE node
W20260810 14:40:54.742686 26306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:54.743122 26140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:54.743777 26140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:54.744937 26140 hybrid_clock.cc:648] HybridClock initialized: now 1786372854744920 us; error 34 us; skew 500 ppm
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:54.747779 26140 init.cc:377] Logged in from keytab as oryx/127.15.84.1@KRBTEST.COM (short username oryx)
I20260810 14:40:54.748694 26140 webserver.cc:460] Webserver started at http://127.15.84.1:40147/ using document root <none> and password file <none>
I20260810 14:40:54.749323 26140 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:54.749418 26140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:54.753701 26140 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:40:54.755892 26320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:54.757444 26140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:54.757556 26140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "aa6e95f155e54ee58454b5767f2ba60d"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "e0e299db0f37b5029446a399cd23cfcd"
server_key_iv: "4707a2fac06e4113cf43f338a74eb954"
server_key_version: "encryptionkey@0"
I20260810 14:40:54.758128 26140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:54.786979 26140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:54.790081 26140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:54.790256 26140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:54.790865 26140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:54.791803 26140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:54.791865 26140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:54.791915 26140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:54.791936 26140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:54.801913 26140 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:33165
I20260810 14:40:54.801957 26493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:33165 every 8 connection(s)
I20260810 14:40:54.803103 26140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:40:54.807188 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 26140
I20260810 14:40:54.807381 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 25277
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.61@KRBTEST.COM
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.60@KRBTEST.COM
I20260810 14:40:54.837416 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.806895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:40023 (local address 127.15.84.62:36385)
0810 14:40:54.807012 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.807016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.820652 (+ 13636us) server_negotiation.cc:407] Connection header received
0810 14:40:54.820861 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.820864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.820914 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.820965 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.822901 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.823587 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.825146 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.825447 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.825511 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.832692 (+ 7181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.832697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.832700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.832734 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.834449 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.835135 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.835140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.835142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.835254 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.835941 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.835945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.835947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.836193 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.836275 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.837035 (+ 760us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.837235 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:40:54.838761 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.806790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.1:40023)
0810 14:40:54.820510 (+ 13720us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.820532 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.820760 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.821473 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.821485 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.822042 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.822723 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.822743 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.823757 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.823760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.825007 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.825019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.825737 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.825740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.825934 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.829933 (+ 3999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.829958 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.832458 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.834566 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.834573 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.834595 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.835011 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.835616 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.835619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.835621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.835826 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.836345 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.836352 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.836730 (+ 378us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.836957 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.837459 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13613,"thread_start_us":88,"threads_started":1}
I20260810 14:40:54.840093 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:35725
--local_ip_for_outbound_sockets=127.15.84.2
--tserver_master_addrs=127.15.84.62:36385
--webserver_port=46245
--webserver_interface=127.15.84.2
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.2.keytab
--principal=oryx/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.84.62:36385,127.15.84.61:36739
--tserver_master_addrs=127.15.84.62:36385,127.15.84.61:36739,127.15.84.60:46403 with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:40:54.840731 26495 heartbeater.cc:347] Connected to a master server at 127.15.84.62:36385
I20260810 14:40:54.840969 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.805065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:39243 (local address 127.15.84.61:36739)
0810 14:40:54.805222 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.805225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.806627 (+ 1402us) server_negotiation.cc:407] Connection header received
0810 14:40:54.807474 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.807478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.807537 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.807603 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.831249 (+ 23646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.832075 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.833248 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.833592 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.833686 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.836072 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.836076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.836079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.836112 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.838507 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.839272 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.839276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.839277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.839370 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.840049 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.840053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.840054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.840291 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.840372 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.840701 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.840830 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:40:54.841022 26495 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:54.841176 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.805646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:58167 (local address 127.15.84.60:46403)
0810 14:40:54.805885 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:54.805889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:54.806298 (+ 409us) server_negotiation.cc:407] Connection header received
0810 14:40:54.807754 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:54.807757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:54.807812 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:54.807894 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:54.831231 (+ 23337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.832303 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.833473 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.833817 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.833895 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.836831 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:54.836835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:54.836837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:54.836871 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:54.838519 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.839102 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.839106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.839108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.839202 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:54.839634 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:54.839637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:54.839639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:54.839891 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:54.839982 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:54.840851 (+ 869us) server_negotiation.cc:299] Negotiation successful
0810 14:40:54.840990 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
I20260810 14:40:54.841529 26503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.805580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:36739 (local address 127.15.84.1:39243)
0810 14:40:54.806526 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.806539 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.807225 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.829856 (+ 22631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.829859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.830252 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.830382 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.830390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.832230 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.832233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.833087 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.833097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.833851 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.833854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.833942 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.834370 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.834386 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.835892 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.838637 (+ 2745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.838640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.838643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.839156 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.839848 (+ 692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.839850 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.839852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.839940 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.840430 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.840433 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.840523 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.841188 (+ 665us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.841340 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":33,"spinlock_wait_cycles":46336,"thread_start_us":176,"threads_started":1}
I20260810 14:40:54.841692 26495 heartbeater.cc:511] Master 127.15.84.62:36385 requested a full tablet report, sending...
I20260810 14:40:54.842325 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:54.805580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:46403 (local address 127.15.84.1:58167)
0810 14:40:54.806152 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:54.806197 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:54.807220 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:54.830741 (+ 23521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:54.830744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:54.830890 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:54.831041 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.831049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.832476 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.832481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.833329 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:54.833341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:54.834091 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:54.834095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:54.834200 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:54.835088 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:54.835103 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:54.836655 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:54.838675 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.838677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.838679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.838987 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.839311 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:54.839314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:54.839316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:54.839393 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:54.840492 (+ 1099us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:54.840496 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:54.840624 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 14:40:54.841805 (+ 1181us) client_negotiation.cc:240] Negotiation successful
0810 14:40:54.842175 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":677,"threads_started":1}
I20260810 14:40:54.842504 26496 heartbeater.cc:347] Connected to a master server at 127.15.84.61:36739
I20260810 14:40:54.842593 26496 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:54.842983 26494 heartbeater.cc:347] Connected to a master server at 127.15.84.60:46403
I20260810 14:40:54.843079 26494 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:54.843235 26494 heartbeater.cc:511] Master 127.15.84.60:46403 requested a full tablet report, sending...
I20260810 14:40:54.843225 24976 ts_manager.cc:194] Re-registered known tserver with Master: aa6e95f155e54ee58454b5767f2ba60d (127.15.84.1:33165)
I20260810 14:40:54.843370 26496 heartbeater.cc:511] Master 127.15.84.61:36739 requested a full tablet report, sending...
I20260810 14:40:54.843995 24976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:40023
I20260810 14:40:54.844409 26219 ts_manager.cc:194] Registered new tserver with Master: aa6e95f155e54ee58454b5767f2ba60d (127.15.84.1:33165)
I20260810 14:40:54.844597 25920 ts_manager.cc:194] Registered new tserver with Master: aa6e95f155e54ee58454b5767f2ba60d (127.15.84.1:33165)
I20260810 14:40:54.912657 25692 heartbeater.cc:503] Master 127.15.84.62:36385 was elected leader, sending a full tablet report...
W20260810 14:40:54.961557 26510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:54.961928 26510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:54.961982 26510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:54.966267 26510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:54.966346 26510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:54.966408 26510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:40:54.972026 26510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.2.keytab
--principal=oryx/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:35725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=46245
--tserver_master_addrs=127.15.84.62:36385,127.15.84.61:36739,127.15.84.60:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:54.973186 26510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:54.974264 26510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:54.984148 26524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:54.984280 26528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:54.984357 26526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:54.984411 26510 server_base.cc:1034] running on GCE node
I20260810 14:40:54.984927 26510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:54.985602 26510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:54.986784 26510 hybrid_clock.cc:648] HybridClock initialized: now 1786372854986771 us; error 33 us; skew 500 ppm
Aug 10 14:40:54 dist-test-slave-rmhg krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:54.989607 26510 init.cc:377] Logged in from keytab as oryx/127.15.84.2@KRBTEST.COM (short username oryx)
I20260810 14:40:54.990535 26510 webserver.cc:460] Webserver started at http://127.15.84.2:46245/ using document root <none> and password file <none>
I20260810 14:40:54.991114 26510 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:54.991201 26510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:54.995368 26510 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:40:54.997741 26549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:54.999171 26510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:54.999301 26510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "f2561cc3377241c0a83657629ac58389"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "9483c4bf5a7a4d6a4427bacc3ea908e9"
server_key_iv: "97def4971b81e2e2f4e726897f08aee2"
server_key_version: "encryptionkey@0"
I20260810 14:40:54.999914 26510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:55.016424 26510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:55.019551 26510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:55.019724 26510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:55.020293 26510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:55.021209 26510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:55.021276 26510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:55.021327 26510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:55.021346 26510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:55.030401 26510 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:35725
I20260810 14:40:55.030443 26741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:35725 every 8 connection(s)
I20260810 14:40:55.031498 26510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:40:55.039247 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 26510
I20260810 14:40:55.040253 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 25486
Aug 10 14:40:55 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
Aug 10 14:40:55 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.61@KRBTEST.COM
Aug 10 14:40:55 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372854, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.60@KRBTEST.COM
I20260810 14:40:55.054325 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.035197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:34831 (local address 127.15.84.62:36385)
0810 14:40:55.035316 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.035319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.035333 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:40:55.036010 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.036014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.036101 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.036307 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:55.038533 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.039382 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.041972 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.042333 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.042433 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.046509 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:55.046514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:55.046518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:55.046568 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:55.051595 (+ 5027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.052305 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.052310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.052312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.052408 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.052890 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.052894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.052896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.053134 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:55.053249 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.054024 (+ 775us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.054161 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:40:55.056209 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.034923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.2:34831)
0810 14:40:55.035194 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:55.035208 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:55.035864 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:55.036567 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:55.036575 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:55.037371 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:55.038332 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.038358 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.039655 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.039658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.041752 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.041768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.042716 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.042720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.042971 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.043753 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:55.043780 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:55.046249 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:55.051743 (+ 5494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:55.051751 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:55.051775 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:55.052177 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:55.052512 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:55.052515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:55.052517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:55.052782 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:55.053322 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:55.053330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:55.053712 (+ 382us) client_negotiation.cc:769] Sending connection context
0810 14:40:55.054559 (+ 847us) client_negotiation.cc:240] Negotiation successful
0810 14:40:55.055009 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":70,"spinlock_wait_cycles":16512,"thread_start_us":90,"threads_started":1}
I20260810 14:40:55.056643 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.049821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:46403 (local address 127.0.0.1:52594)
0810 14:40:55.050132 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:55.050158 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:55.050245 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:55.052195 (+ 1950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:55.052198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:55.052220 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:55.052323 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.052331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.053441 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.053445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.054604 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.054618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.056258 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.056262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.056361 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.056389 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 14:40:55.056438 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 14:40:55.056525 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260810 14:40:55.057622 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.033908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:36739 (local address 127.15.84.2:35649)
0810 14:40:55.034771 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:55.034784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:55.035880 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:55.036544 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:55.036554 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:55.037366 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:55.038360 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.038373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.040022 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.040025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.043603 (+ 3578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.043618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.044335 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.044338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.044417 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.048297 (+ 3880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:55.048313 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:55.050267 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:55.055678 (+ 5411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:55.055681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:55.055683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:55.055915 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:55.056243 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:55.056246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:55.056248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:55.056341 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:55.056824 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:55.056827 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:55.056919 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 14:40:55.056998 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 14:40:55.057131 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":174,"spinlock_wait_cycles":118272,"thread_start_us":124,"threads_started":1}
I20260810 14:40:55.058125 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.033458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:35649 (local address 127.15.84.61:36739)
0810 14:40:55.033607 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.033612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.034880 (+ 1268us) server_negotiation.cc:407] Connection header received
0810 14:40:55.036051 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.036055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.036117 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.036182 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:55.038575 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.039363 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.043781 (+ 4418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.044119 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.044191 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.050448 (+ 6257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:55.050452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:55.050455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:55.050488 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:55.055541 (+ 5053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.056038 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.056042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.056044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.056143 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.056442 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.056445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.056447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.056687 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:55.056780 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.057846 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.057978 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:40:55.058162 26743 heartbeater.cc:347] Connected to a master server at 127.15.84.62:36385
I20260810 14:40:55.058434 26743 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:55.059082 26743 heartbeater.cc:511] Master 127.15.84.62:36385 requested a full tablet report, sending...
I20260810 14:40:55.059981 26745 heartbeater.cc:347] Connected to a master server at 127.15.84.61:36739
I20260810 14:40:55.060089 26745 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:55.060209 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:46843
--local_ip_for_outbound_sockets=127.15.84.3
--tserver_master_addrs=127.15.84.62:36385
--webserver_port=45575
--webserver_interface=127.15.84.3
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.3.keytab
--principal=oryx/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.84.62:36385,127.15.84.61:36739
--tserver_master_addrs=127.15.84.62:36385,127.15.84.61:36739,127.15.84.60:46403 with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:40:55.060261 26745 heartbeater.cc:511] Master 127.15.84.61:36739 requested a full tablet report, sending...
I20260810 14:40:55.061025 25920 ts_manager.cc:194] Registered new tserver with Master: f2561cc3377241c0a83657629ac58389 (127.15.84.2:35725)
I20260810 14:40:55.061048 24976 ts_manager.cc:194] Re-registered known tserver with Master: f2561cc3377241c0a83657629ac58389 (127.15.84.2:35725)
I20260810 14:40:55.061798 24976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:34831
I20260810 14:40:55.062507 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.051679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.15.84.60:46403)
0810 14:40:55.051940 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.051944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.051956 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:40:55.052000 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.052002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.052055 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.052154 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:55.052466 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.053302 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.055397 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.056110 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.057453 (+ 1343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.057960 (+ 507us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.058001 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.058080 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":103,"threads_started":1}
I20260810 14:40:55.062763 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.033457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:51273 (local address 127.15.84.60:46403)
0810 14:40:55.033616 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.033621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.034622 (+ 1001us) server_negotiation.cc:407] Connection header received
0810 14:40:55.036108 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.036129 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.036190 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.036242 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:55.038580 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.040019 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.049498 (+ 9479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.049854 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.049946 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.055006 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:55.055011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:55.055014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:55.055048 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:55.060305 (+ 5257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.060819 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.060822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.060824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.060921 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.061407 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.061410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.061412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.061626 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:55.061697 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.062467 (+ 770us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.062615 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:40:55.063120 26238 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 14:40:55.063449 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.033908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:46403 (local address 127.15.84.2:51273)
0810 14:40:55.034432 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:55.034496 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:55.035655 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:55.036544 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:55.036554 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:55.037538 (+ 984us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:55.038379 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.038386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.048420 (+ 10034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.048423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.049327 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.049340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.050412 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.050416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.050510 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.052504 (+ 1994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:55.052519 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:55.054005 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:55.060437 (+ 6432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:55.060440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:55.060442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:55.060696 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:55.061047 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:55.061051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:55.061053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:55.061175 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:55.062151 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:55.062154 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:55.062275 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 14:40:55.063077 (+ 802us) client_negotiation.cc:240] Negotiation successful
0810 14:40:55.063240 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":175,"spinlock_wait_cycles":9344,"thread_start_us":146,"threads_started":1}
I20260810 14:40:55.063568 26289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2767b706cceb48a5bf5440ff912fdea8. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } } }
I20260810 14:40:55.063714 26289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: This master's current role is: LEARNER
I20260810 14:40:55.064347 26742 heartbeater.cc:347] Connected to a master server at 127.15.84.60:46403
I20260810 14:40:55.064432 26742 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:55.064649 26742 heartbeater.cc:511] Master 127.15.84.60:46403 requested a full tablet report, sending...
I20260810 14:40:55.065347 26219 ts_manager.cc:194] Registered new tserver with Master: f2561cc3377241c0a83657629ac58389 (127.15.84.2:35725)
W20260810 14:40:55.183034 26755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:55.183332 26755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:55.183394 26755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:55.187603 26755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:55.187683 26755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:55.187745 26755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:40:55.193324 26755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.3.keytab
--principal=oryx/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:46843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=45575
--tserver_master_addrs=127.15.84.62:36385,127.15.84.61:36739,127.15.84.60:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:55.194490 26755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:55.195554 26755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:55.206503 26763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:55.206553 26755 server_base.cc:1034] running on GCE node
W20260810 14:40:55.206486 26769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:55.206485 26765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:55.207334 26755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:55.208045 26755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:55.209235 26755 hybrid_clock.cc:648] HybridClock initialized: now 1786372855209208 us; error 37 us; skew 500 ppm
Aug 10 14:40:55 dist-test-slave-rmhg krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372855, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:55.212155 26755 init.cc:377] Logged in from keytab as oryx/127.15.84.3@KRBTEST.COM (short username oryx)
I20260810 14:40:55.213135 26755 webserver.cc:460] Webserver started at http://127.15.84.3:45575/ using document root <none> and password file <none>
I20260810 14:40:55.213758 26755 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:55.213845 26755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:55.218148 26755 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 14:40:55.220487 26778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:55.221870 26755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:40:55.221984 26755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "9327ffe3276c48a49fa54211a6e32fff"
format_stamp: "Formatted at 2026-08-10 14:40:53 on dist-test-slave-rmhg"
server_key: "89230b6c35b63ba6d0fb3fad5cf4aa03"
server_key_iv: "0850bca9e040c901331375a60e9e3ce5"
server_key_version: "encryptionkey@0"
I20260810 14:40:55.222572 26755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:55.247530 26755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:55.250954 26755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:55.251153 26755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:55.251788 26755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:40:55.252684 26755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:55.252732 26755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:55.252777 26755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:55.252806 26755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:55.261770 26755 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:46843
I20260810 14:40:55.261796 26957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:46843 every 8 connection(s)
I20260810 14:40:55.262933 26755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:40:55.269459 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 26755
Aug 10 14:40:55 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372855, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.60@KRBTEST.COM
Aug 10 14:40:55 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372855, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.61@KRBTEST.COM
Aug 10 14:40:55 dist-test-slave-rmhg krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372855, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
I20260810 14:40:55.282598 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.264875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:45529 (local address 127.15.84.62:36385)
0810 14:40:55.265023 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.265027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.266248 (+ 1221us) server_negotiation.cc:407] Connection header received
0810 14:40:55.267457 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.267461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.267519 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.267570 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:55.269250 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.270028 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.273502 (+ 3474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.273837 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.273958 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.276978 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:55.276982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:55.276985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:55.277016 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:55.279729 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.280551 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.280555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.280557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.280650 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.281079 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.281082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.281084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.281374 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:55.281471 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.282289 (+ 818us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.282418 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:40:55.283501 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.266805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:56067 (local address 127.15.84.61:36739)
0810 14:40:55.266933 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.266938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.267010 (+ 72us) server_negotiation.cc:407] Connection header received
0810 14:40:55.267427 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.267431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.267488 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.267561 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:55.270046 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.270804 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.272469 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.272792 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.272868 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.276849 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:55.276854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:55.276856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:55.276890 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:55.278919 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.280644 (+ 1725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.280648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.280650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.280756 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.281479 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.281483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.281485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.281719 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:55.281789 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.283191 (+ 1402us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.283351 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:40:55.283944 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.271423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.15.84.62:36385)
0810 14:40:55.271681 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.271695 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.272397 (+ 702us) server_negotiation.cc:407] Connection header received
0810 14:40:55.272601 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.272604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.272659 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.272722 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:55.275918 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.276706 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.277849 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.278214 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.278296 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.279938 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:55.279941 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:55.279944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:55.279971 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:55.281685 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.282278 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.282282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.282283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.282374 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.282865 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.282868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.282870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.283078 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:55.283222 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.283655 (+ 433us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.283787 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
I20260810 14:40:55.284629 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.264950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:54311 (local address 127.15.84.60:46403)
0810 14:40:55.265060 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.265063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.266248 (+ 1185us) server_negotiation.cc:407] Connection header received
0810 14:40:55.267458 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.267462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.267519 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.267571 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:55.269020 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.269864 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.271705 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.272024 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.272112 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.277307 (+ 5195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:55.277311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:55.277314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:55.277345 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:55.279727 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.281262 (+ 1535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.281265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.281267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.281359 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.282371 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.282374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.282376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.282645 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:55.282740 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.284310 (+ 1570us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.284454 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46}
I20260810 14:40:55.285604 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.265502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.3:45529)
0810 14:40:55.266066 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:55.266131 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:55.267220 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:55.267746 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:55.267754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:55.268221 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:55.268855 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.268871 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.270161 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.270165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.272317 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.272329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.274079 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.274083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.274192 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.275285 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:55.275301 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:55.276801 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:55.279881 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:55.279884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:55.279895 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:55.280436 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:55.280758 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:55.280760 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:55.280762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:55.280970 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:55.281568 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:55.281575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:55.281969 (+ 394us) client_negotiation.cc:769] Sending connection context
0810 14:40:55.283119 (+ 1150us) client_negotiation.cc:240] Negotiation successful
0810 14:40:55.283544 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":1091,"spinlock_wait_cycles":93952,"thread_start_us":150,"threads_started":1}
I20260810 14:40:55.287318 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.265502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:46403 (local address 127.15.84.3:54311)
0810 14:40:55.266066 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:55.266131 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:55.267220 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:55.267746 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:55.267754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:55.268222 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:55.268855 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.268871 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.270036 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.270041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.271534 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.271546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.272566 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.272571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.272858 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.273566 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:55.273588 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:55.276235 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:55.280098 (+ 3863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:55.280101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:55.280103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:55.281115 (+ 1012us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:55.281861 (+ 746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:55.281864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:55.281866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:55.282269 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:55.282893 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:55.282895 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:55.282988 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 14:40:55.284823 (+ 1835us) client_negotiation.cc:240] Negotiation successful
0810 14:40:55.284966 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":140,"spinlock_wait_cycles":19200,"thread_start_us":146,"threads_started":1}
I20260810 14:40:55.285609 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.266672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:36739 (local address 127.15.84.3:56067)
0810 14:40:55.266930 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:55.266943 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:55.267219 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:55.267744 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:55.267752 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:55.268218 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:55.268854 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.268870 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.270918 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.270921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.271712 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.271724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.273028 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.273031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.273166 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.273670 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:55.273685 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:55.276652 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:55.279861 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:55.279869 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:55.279893 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:55.280518 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:55.280877 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:55.280880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:55.280882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:55.280979 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:55.281954 (+ 975us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:55.281956 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:55.282049 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 14:40:55.283720 (+ 1671us) client_negotiation.cc:240] Negotiation successful
0810 14:40:55.283863 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":72,"thread_start_us":76,"threads_started":1}
I20260810 14:40:55.287350 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.271889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.15.84.61:36739)
0810 14:40:55.272135 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.272139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.272821 (+ 682us) server_negotiation.cc:407] Connection header received
0810 14:40:55.272868 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.272871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.272919 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.273000 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:55.274522 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.275134 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.277311 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.277680 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.277782 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.281104 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:55.281107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:55.281110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:55.281135 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:55.283755 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.285091 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.285094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.285096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.285234 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.285842 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.285845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.285847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.286596 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:55.286729 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.287044 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.287186 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
I20260810 14:40:55.288722 26961 heartbeater.cc:347] Connected to a master server at 127.15.84.61:36739
I20260810 14:40:55.288744 26960 heartbeater.cc:347] Connected to a master server at 127.15.84.62:36385
I20260810 14:40:55.289009 26960 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:55.289008 26961 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:55.289716 26960 heartbeater.cc:511] Master 127.15.84.62:36385 requested a full tablet report, sending...
I20260810 14:40:55.289716 26961 heartbeater.cc:511] Master 127.15.84.61:36739 requested a full tablet report, sending...
I20260810 14:40:55.287725 24975 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 14:40:55.290807 24976 ts_manager.cc:194] Re-registered known tserver with Master: 9327ffe3276c48a49fa54211a6e32fff (127.15.84.3:46843)
I20260810 14:40:55.291047 25920 ts_manager.cc:194] Registered new tserver with Master: 9327ffe3276c48a49fa54211a6e32fff (127.15.84.3:46843)
I20260810 14:40:55.291531 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.272312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.15.84.60:46403)
0810 14:40:55.272433 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.272437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.273114 (+ 677us) server_negotiation.cc:407] Connection header received
0810 14:40:55.276203 (+ 3089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.276207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.276273 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.276330 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:55.278284 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.279075 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.282095 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.282420 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.282505 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.285874 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:55.285878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:55.285880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:55.285907 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:55.287386 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.288382 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.288386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.288388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.288473 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.290459 (+ 1986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.290463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.290464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.290723 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:55.290860 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.291228 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.291376 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":263,"server-negotiator.queue_time_us":56}
I20260810 14:40:55.291508 24976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:45529
W20260810 14:40:55.291774 24975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:55.295280 26959 heartbeater.cc:347] Connected to a master server at 127.15.84.60:46403
I20260810 14:40:55.295414 26959 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:55.295652 26959 heartbeater.cc:511] Master 127.15.84.60:46403 requested a full tablet report, sending...
I20260810 14:40:55.296458 26219 ts_manager.cc:194] Registered new tserver with Master: 9327ffe3276c48a49fa54211a6e32fff (127.15.84.3:46843)
I20260810 14:40:55.296451 26989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: NON_VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: true } } }
I20260810 14:40:55.296578 26989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:55.298317 26989 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8: attempting to promote NON_VOTER 3b9146920cbc4e7a9dd800bb40b11d7d to VOTER
I20260810 14:40:55.298659 26989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } }
I20260810 14:40:55.299990 26238 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 LEARNER]: Refusing update from remote peer 2767b706cceb48a5bf5440ff912fdea8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 14:40:55.299988 25932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 FOLLOWER]: Refusing update from remote peer 2767b706cceb48a5bf5440ff912fdea8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 14:40:55.300400 26999 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 14:40:55.300555 26989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 14:40:55.302335 26162 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3b9146920cbc4e7a9dd800bb40b11d7d (127.15.84.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } } }
I20260810 14:40:55.302659 25932 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3b9146920cbc4e7a9dd800bb40b11d7d (127.15.84.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } } }
I20260810 14:40:55.302672 26238 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3b9146920cbc4e7a9dd800bb40b11d7d (127.15.84.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } } }
I20260810 14:40:55.303872 26162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } } }
I20260810 14:40:55.303992 26162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:55.304141 26162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } } }
I20260810 14:40:55.304225 26162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:55.304322 26162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } } }
I20260810 14:40:55.304314 26289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } } }
I20260810 14:40:55.304399 26162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:55.304409 26289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:55.304648 27004 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2767b706cceb48a5bf5440ff912fdea8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } } }
I20260810 14:40:55.304798 27004 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:55.317112 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.311583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:35725 (local address 127.0.0.1:33688)
0810 14:40:55.311727 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:55.311744 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:55.311828 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:55.312702 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:55.312705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:55.312723 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:55.312856 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.312863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.314522 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.314525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.315699 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.315710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.316679 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.316683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.316803 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.316831 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 14:40:55.316908 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:40:55.316987 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":1,"spinlock_wait_cycles":768}
I20260810 14:40:55.317615 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.312365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:33165 (local address 127.0.0.1:35240)
0810 14:40:55.312959 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:55.312971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:55.313076 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:55.313621 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:55.313624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:55.313642 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:55.313740 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.313747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.314947 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.314951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.316135 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.316143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.317243 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.317247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.317336 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.317362 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:40:55.317410 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 14:40:55.317482 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":188,"thread_start_us":99,"threads_started":1}
I20260810 14:40:55.318234 27010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.312806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.15.84.1:33165)
0810 14:40:55.313119 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.313170 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.313195 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:40:55.313275 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.313280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.313470 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.313632 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:55.313833 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.314824 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.316271 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.317091 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.317495 (+ 404us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.317860 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.317934 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.318073 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260810 14:40:55.318454 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.311844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:46843 (local address 127.0.0.1:33010)
0810 14:40:55.312672 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:55.312686 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:55.312789 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:55.313807 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:55.313810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:55.313827 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:55.313904 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.313908 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.315039 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.315043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.316834 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.316841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.318065 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.318069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.318153 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.318185 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 14:40:55.318250 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 14:40:55.318322 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":877,"spinlock_wait_cycles":1280,"thread_start_us":110,"threads_started":1}
I20260810 14:40:55.318521 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.312268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.15.84.3:46843)
0810 14:40:55.313275 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.313285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.313305 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:40:55.313369 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.313373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.313523 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.313657 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:55.314009 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.314910 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.316935 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.317953 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.318088 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.318231 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.318287 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.318389 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":121,"threads_started":1}
I20260810 14:40:55.319706 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.311779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.15.84.2:35725)
0810 14:40:55.312121 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.312134 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.312158 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:40:55.312248 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.312254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.312517 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.313019 (+ 502us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:55.313209 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.314398 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.315823 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.316541 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.316665 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.316840 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.319434 (+ 2594us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.319550 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":142,"threads_started":1}
I20260810 14:40:55.321600 26404 tablet_service.cc:1467] Processing CreateTablet for tablet 8d60dd4026c14b17978b4e9530aeb1fc (DEFAULT_TABLE table=test-table [id=b24a19f29a1f465198696a3e6d9854da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:55.321600 26850 tablet_service.cc:1467] Processing CreateTablet for tablet 8d60dd4026c14b17978b4e9530aeb1fc (DEFAULT_TABLE table=test-table [id=b24a19f29a1f465198696a3e6d9854da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:55.322834 26404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d60dd4026c14b17978b4e9530aeb1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:55.322834 26850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d60dd4026c14b17978b4e9530aeb1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:55.322561 26626 tablet_service.cc:1467] Processing CreateTablet for tablet 8d60dd4026c14b17978b4e9530aeb1fc (DEFAULT_TABLE table=test-table [id=b24a19f29a1f465198696a3e6d9854da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:55.323565 26626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d60dd4026c14b17978b4e9530aeb1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:55.332508 27013 tablet_bootstrap.cc:492] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389: Bootstrap starting.
I20260810 14:40:55.332863 27011 tablet_bootstrap.cc:492] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff: Bootstrap starting.
I20260810 14:40:55.335213 27011 tablet_bootstrap.cc:654] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:55.335235 27013 tablet_bootstrap.cc:654] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:55.336195 27011 log.cc:824] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:55.336195 27013 log.cc:824] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:55.336285 27014 tablet_bootstrap.cc:492] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d: Bootstrap starting.
I20260810 14:40:55.338627 27014 tablet_bootstrap.cc:654] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:55.338989 27011 tablet_bootstrap.cc:492] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff: No bootstrap required, opened a new log
I20260810 14:40:55.339003 27013 tablet_bootstrap.cc:492] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389: No bootstrap required, opened a new log
I20260810 14:40:55.339202 27013 ts_tablet_manager.cc:1397] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 14:40:55.339202 27011 ts_tablet_manager.cc:1397] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:40:55.339531 27014 log.cc:824] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:55.341609 27014 tablet_bootstrap.cc:492] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d: No bootstrap required, opened a new log
I20260810 14:40:55.341815 27014 ts_tablet_manager.cc:1397] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260810 14:40:55.344046 27011 raft_consensus.cc:348] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } }
I20260810 14:40:55.344041 27013 raft_consensus.cc:348] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } }
I20260810 14:40:55.344357 27011 raft_consensus.cc:374] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:55.344357 27013 raft_consensus.cc:374] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:55.344417 27013 raft_consensus.cc:729] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2561cc3377241c0a83657629ac58389, State: Initialized, Role: FOLLOWER
I20260810 14:40:55.344417 27011 raft_consensus.cc:729] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9327ffe3276c48a49fa54211a6e32fff, State: Initialized, Role: FOLLOWER
I20260810 14:40:55.344815 27014 raft_consensus.cc:348] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } }
I20260810 14:40:55.344951 27011 consensus_queue.cc:260] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } }
I20260810 14:40:55.344951 27013 consensus_queue.cc:260] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } }
I20260810 14:40:55.345096 27014 raft_consensus.cc:374] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:55.345175 27014 raft_consensus.cc:729] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6e95f155e54ee58454b5767f2ba60d, State: Initialized, Role: FOLLOWER
I20260810 14:40:55.345738 27014 consensus_queue.cc:260] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } }
I20260810 14:40:55.346088 27013 ts_tablet_manager.cc:1428] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 14:40:55.346451 26495 heartbeater.cc:503] Master 127.15.84.62:36385 was elected leader, sending a full tablet report...
I20260810 14:40:55.346709 27014 ts_tablet_manager.cc:1428] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 14:40:55.346783 26743 heartbeater.cc:503] Master 127.15.84.62:36385 was elected leader, sending a full tablet report...
I20260810 14:40:55.347371 26960 heartbeater.cc:503] Master 127.15.84.62:36385 was elected leader, sending a full tablet report...
I20260810 14:40:55.352563 27011 ts_tablet_manager.cc:1428] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260810 14:40:55.426407 27022 raft_consensus.cc:482] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:55.426653 27022 raft_consensus.cc:504] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } }
I20260810 14:40:55.427937 27022 leader_election.cc:288] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9327ffe3276c48a49fa54211a6e32fff (127.15.84.3:46843), aa6e95f155e54ee58454b5767f2ba60d (127.15.84.1:33165)
I20260810 14:40:55.431730 27010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.428293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:51369 (local address 127.15.84.1:33165)
0810 14:40:55.428433 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.428438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.428525 (+ 87us) server_negotiation.cc:407] Connection header received
0810 14:40:55.428662 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.428665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.428715 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.428787 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:55.429036 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.429601 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.430726 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.431115 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.431212 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.431283 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.431531 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.431607 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:40:55.431730 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.428293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:51553 (local address 127.15.84.3:46843)
0810 14:40:55.428433 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.428437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.428496 (+ 59us) server_negotiation.cc:407] Connection header received
0810 14:40:55.428643 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.428646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.428697 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.428757 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:55.429026 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.429591 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.430727 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.431117 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.431220 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.431286 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.431528 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.431601 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:40:55.431748 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.428193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:33165 (local address 127.15.84.2:51369)
0810 14:40:55.428436 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:55.428450 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:55.428562 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:55.428812 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:55.428815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:55.428833 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:55.428933 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.428937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.429711 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.429714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.430593 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.430604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.431236 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.431240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.431413 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.431436 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:40:55.431506 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 14:40:55.431580 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":103}
I20260810 14:40:55.431756 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.428193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:46843 (local address 127.15.84.2:51553)
0810 14:40:55.428415 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:55.428430 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:55.428539 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:55.428786 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:55.428789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:55.428805 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:55.428926 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.428932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.429706 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.429709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.430597 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:55.430607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.431237 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.431240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:55.431400 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.431428 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 14:40:55.431491 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 14:40:55.431570 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":88}
I20260810 14:40:55.432449 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d60dd4026c14b17978b4e9530aeb1fc" candidate_uuid: "f2561cc3377241c0a83657629ac58389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6e95f155e54ee58454b5767f2ba60d" is_pre_election: true
I20260810 14:40:55.432771 26881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d60dd4026c14b17978b4e9530aeb1fc" candidate_uuid: "f2561cc3377241c0a83657629ac58389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9327ffe3276c48a49fa54211a6e32fff" is_pre_election: true
I20260810 14:40:55.432842 26427 raft_consensus.cc:2454] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2561cc3377241c0a83657629ac58389 in term 0.
I20260810 14:40:55.433065 26881 raft_consensus.cc:2454] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2561cc3377241c0a83657629ac58389 in term 0.
I20260810 14:40:55.433462 26556 leader_election.cc:302] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa6e95f155e54ee58454b5767f2ba60d, f2561cc3377241c0a83657629ac58389; no voters:
I20260810 14:40:55.433774 27022 raft_consensus.cc:2781] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:40:55.433871 27022 raft_consensus.cc:482] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:40:55.433923 27022 raft_consensus.cc:3037] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:55.435010 27022 raft_consensus.cc:504] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } }
I20260810 14:40:55.435429 27022 leader_election.cc:288] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [CANDIDATE]: Term 1 election: Requested vote from peers 9327ffe3276c48a49fa54211a6e32fff (127.15.84.3:46843), aa6e95f155e54ee58454b5767f2ba60d (127.15.84.1:33165)
I20260810 14:40:55.435819 26881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d60dd4026c14b17978b4e9530aeb1fc" candidate_uuid: "f2561cc3377241c0a83657629ac58389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9327ffe3276c48a49fa54211a6e32fff"
I20260810 14:40:55.435917 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d60dd4026c14b17978b4e9530aeb1fc" candidate_uuid: "f2561cc3377241c0a83657629ac58389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6e95f155e54ee58454b5767f2ba60d"
I20260810 14:40:55.435966 26881 raft_consensus.cc:3037] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:55.436048 26427 raft_consensus.cc:3037] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:55.437429 26881 raft_consensus.cc:2454] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2561cc3377241c0a83657629ac58389 in term 1.
I20260810 14:40:55.437429 26427 raft_consensus.cc:2454] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2561cc3377241c0a83657629ac58389 in term 1.
I20260810 14:40:55.437754 26556 leader_election.cc:302] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9327ffe3276c48a49fa54211a6e32fff, aa6e95f155e54ee58454b5767f2ba60d, f2561cc3377241c0a83657629ac58389; no voters:
I20260810 14:40:55.437947 27022 raft_consensus.cc:2781] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:55.438303 27022 raft_consensus.cc:686] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [term 1 LEADER]: Becoming Leader. State: Replica: f2561cc3377241c0a83657629ac58389, State: Running, Role: LEADER
I20260810 14:40:55.438726 27022 consensus_queue.cc:237] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } }
I20260810 14:40:55.442088 24973 catalog_manager.cc:5495] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2561cc3377241c0a83657629ac58389 (127.15.84.2). New cstate: current_term: 1 leader_uuid: "f2561cc3377241c0a83657629ac58389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } health_report { overall_health: UNKNOWN } } }
I20260810 14:40:55.481452 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.478142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33694 (local address 127.15.84.2:35725)
0810 14:40:55.478264 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.478267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.478281 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:40:55.478366 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.478369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.478429 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.478522 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:55.478735 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.479246 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.480071 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.480300 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.480379 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.480631 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:55.481098 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:55.481238 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.481268 (+ 30us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.481329 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:40:55.488080 26881 raft_consensus.cc:1261] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff [term 1 FOLLOWER]: Refusing update from remote peer f2561cc3377241c0a83657629ac58389: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:55.488080 26427 raft_consensus.cc:1261] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d [term 1 FOLLOWER]: Refusing update from remote peer f2561cc3377241c0a83657629ac58389: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:55.489024 27026 consensus_queue.cc:1035] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:55.489253 27022 consensus_queue.cc:1035] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:55.498184 27029 mvcc.cc:205] Tried to move back new op lower bound from 7316983216070692864 to 7316983215880060928. Current Snapshot: MvccSnapshot[applied={T|T < 7316983216070692864}]
I20260810 14:40:55.504287 27031 mvcc.cc:205] Tried to move back new op lower bound from 7316983216070692864 to 7316983215880060928. Current Snapshot: MvccSnapshot[applied={T|T < 7316983216070692864}]
I20260810 14:40:55.505010 27030 mvcc.cc:205] Tried to move back new op lower bound from 7316983216070692864 to 7316983215880060928. Current Snapshot: MvccSnapshot[applied={T|T < 7316983216070692864 or (T in {7316983216070692864})}]
I20260810 14:40:55.509266 24973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:55.509429 24973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:55.514122 24973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2767b706cceb48a5bf5440ff912fdea8: Sending DeleteTablet for 3 replicas of tablet 8d60dd4026c14b17978b4e9530aeb1fc
I20260810 14:40:55.515560 26626 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d60dd4026c14b17978b4e9530aeb1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:55 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:33688
I20260810 14:40:55.515712 26850 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d60dd4026c14b17978b4e9530aeb1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:55 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:33010
I20260810 14:40:55.515990 26404 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d60dd4026c14b17978b4e9530aeb1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:55 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:35240
I20260810 14:40:55.516187 27043 tablet_replica.cc:330] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff: stopping tablet replica
I20260810 14:40:55.516564 27043 raft_consensus.cc:2229] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:55.516772 27045 tablet_replica.cc:330] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d: stopping tablet replica
I20260810 14:40:55.517092 27045 raft_consensus.cc:2229] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:55.517513 27045 raft_consensus.cc:2258] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:55.518635 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 24927
I20260810 14:40:55.527225 27045 ts_tablet_manager.cc:1905] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:55.530463 27045 ts_tablet_manager.cc:1918] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:55.530591 27045 log.cc:1196] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-0/wal/wals/8d60dd4026c14b17978b4e9530aeb1fc
I20260810 14:40:55.530921 27045 ts_tablet_manager.cc:1939] T 8d60dd4026c14b17978b4e9530aeb1fc P aa6e95f155e54ee58454b5767f2ba60d: Deleting consensus metadata
I20260810 14:40:55.532821 27043 raft_consensus.cc:2258] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:55.532898 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.517394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.15.84.61:36739)
0810 14:40:55.517515 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.517519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.518388 (+ 869us) server_negotiation.cc:407] Connection header received
0810 14:40:55.518579 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.518583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.518654 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.518712 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:55.522222 (+ 3510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.523291 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.524826 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.525301 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.525402 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.528369 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:55.528375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:55.528378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:55.528418 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:55.530452 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.531127 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.531132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.531134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.531256 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.531653 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.531657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.531659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.531945 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:55.532064 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.532518 (+ 454us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.532710 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:40:55.533324 27044 tablet_replica.cc:330] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389: stopping tablet replica
I20260810 14:40:55.535313 27043 ts_tablet_manager.cc:1905] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:55.535904 27044 raft_consensus.cc:2229] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:40:55.536489 27044 raft_consensus.cc:2258] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:55.539897 27044 ts_tablet_manager.cc:1905] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:55.542985 27044 ts_tablet_manager.cc:1918] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:55.543123 27044 log.cc:1196] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/8d60dd4026c14b17978b4e9530aeb1fc
I20260810 14:40:55.543251 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.517938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.15.84.60:46403)
0810 14:40:55.518043 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.518047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.533658 (+ 15611us) server_negotiation.cc:407] Connection header received
0810 14:40:55.533815 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.533820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.533879 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.533931 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:55.534830 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.535658 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.537192 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.537528 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.537606 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.539458 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:55.539462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:55.539465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:55.539496 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:55.541236 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.541776 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.541780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.541782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.541883 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.542197 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.542200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.542202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.542458 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:55.542569 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.542950 (+ 381us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.543088 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 14:40:55.543460 27044 ts_tablet_manager.cc:1939] T 8d60dd4026c14b17978b4e9530aeb1fc P f2561cc3377241c0a83657629ac58389: Deleting consensus metadata
I20260810 14:40:55.545598 27043 ts_tablet_manager.cc:1918] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:55.545768 27043 log.cc:1196] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/8d60dd4026c14b17978b4e9530aeb1fc
I20260810 14:40:55.546111 27043 ts_tablet_manager.cc:1939] T 8d60dd4026c14b17978b4e9530aeb1fc P 9327ffe3276c48a49fa54211a6e32fff: Deleting consensus metadata
W20260810 14:40:55.549346 26782 connection.cc:535] server connection from 127.0.0.1:33010 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 14:40:55.549350 26553 connection.cc:535] server connection from 127.0.0.1:33688 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 14:40:55.549368 26323 connection.cc:535] server connection from 127.0.0.1:35240 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 14:40:55.561442 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.553011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.15.84.60:46403)
0810 14:40:55.553190 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.553195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.553527 (+ 332us) server_negotiation.cc:407] Connection header received
0810 14:40:55.553748 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.553752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.553807 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.553900 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:55.554762 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.555296 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.556171 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.556407 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.556502 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.558001 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:55.558004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:55.558006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:55.558035 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:55.559468 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.559885 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.559889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.559891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.559975 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.560242 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.560246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.560248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.560510 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:55.560649 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.560998 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.561225 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:40:55.563338 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:55.553430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57658 (local address 127.15.84.61:36739)
0810 14:40:55.553548 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:55.553552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:55.554171 (+ 619us) server_negotiation.cc:407] Connection header received
0810 14:40:55.554277 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:55.554280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:55.554339 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:55.554426 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:55.555281 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.556120 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.557721 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:55.558074 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:55.558151 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:55.559834 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:55.559838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:55.559841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:55.559874 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:55.561447 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.561919 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.561922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.561924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.562020 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:55.562373 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:55.562377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:55.562379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:55.562638 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:55.562772 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:55.563053 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 14:40:55.563178 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:40:56.501623 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:56.501121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.2:34041)
0810 14:40:56.501433 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:56.501523 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":82,"threads_started":1}
W20260810 14:40:56.501977 26743 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:36385 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
I20260810 14:40:56.504354 27059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:56.503869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.1:57599)
0810 14:40:56.504140 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:56.504243 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
W20260810 14:40:56.504722 26495 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:36385 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
I20260810 14:40:56.505991 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:56.505544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.3:60407)
0810 14:40:56.505804 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:56.505900 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
W20260810 14:40:56.506284 26960 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:36385 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
I20260810 14:40:57.014853 27067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2767b706cceb48a5bf5440ff912fdea8)
I20260810 14:40:57.014868 27066 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2767b706cceb48a5bf5440ff912fdea8)
I20260810 14:40:57.015008 27067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } }
I20260810 14:40:57.015061 27066 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } }
I20260810 14:40:57.015560 27067 leader_election.cc:288] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2767b706cceb48a5bf5440ff912fdea8 (127.15.84.62:36385), ee49a0d980e24a6a9d712deafedd3091 (127.15.84.61:36739)
I20260810 14:40:57.016391 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.015775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.0.0.1:44384)
0810 14:40:57.016189 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.016285 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":55,"threads_started":1}
W20260810 14:40:57.016602 26189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
I20260810 14:40:57.017017 27066 leader_election.cc:288] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2767b706cceb48a5bf5440ff912fdea8 (127.15.84.62:36385), 3b9146920cbc4e7a9dd800bb40b11d7d (127.15.84.60:46403)
I20260810 14:40:57.017278 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.016214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.0.0.1:44388)
0810 14:40:57.016426 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.017182 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
W20260810 14:40:57.017448 25899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
I20260810 14:40:57.017728 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.017479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.0.0.1:44392)
0810 14:40:57.017606 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.017657 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260810 14:40:57.017863 26189 leader_election.cc:334] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2767b706cceb48a5bf5440ff912fdea8 (127.15.84.62:36385): Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
I20260810 14:40:57.018635 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.018402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.0.0.1:44394)
0810 14:40:57.018518 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.018568 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 14:40:57.018839 25899 leader_election.cc:334] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2767b706cceb48a5bf5440ff912fdea8 (127.15.84.62:36385): Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
I20260810 14:40:57.020354 27072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.016434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.15.84.61:36739)
0810 14:40:57.016676 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:57.016681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:57.016721 (+ 40us) server_negotiation.cc:407] Connection header received
0810 14:40:57.016773 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:57.016776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:57.016826 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:57.016905 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:57.017394 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.018125 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.019415 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.019844 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.019937 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.020001 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:57.020158 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 14:40:57.020238 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":60,"threads_started":1}
I20260810 14:40:57.020395 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.016083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:36739 (local address 127.0.0.1:57670)
0810 14:40:57.016345 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.016361 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:57.016619 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:57.017036 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:57.017040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:57.017058 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:57.017259 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:57.017267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.018228 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.018232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:57.019301 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:57.019311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.019954 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.019958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:57.020048 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.020069 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 14:40:57.020134 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 14:40:57.020255 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":50,"thread_start_us":83,"threads_started":1}
I20260810 14:40:57.020752 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ee49a0d980e24a6a9d712deafedd3091" is_pre_election: true
I20260810 14:40:57.020900 25932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9146920cbc4e7a9dd800bb40b11d7d in term 1.
I20260810 14:40:57.021239 27073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.016835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:46403 (local address 127.0.0.1:52628)
0810 14:40:57.017593 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.017606 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:57.017708 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:57.018125 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:57.018128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:57.018145 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:57.018236 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:57.018244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.019171 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.019175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:57.019987 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:57.019996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.020820 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.020824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:57.020912 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.020930 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 14:40:57.020996 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 14:40:57.021066 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":5,"thread_start_us":80,"threads_started":1}
I20260810 14:40:57.021211 26189 leader_election.cc:302] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b9146920cbc4e7a9dd800bb40b11d7d, ee49a0d980e24a6a9d712deafedd3091; no voters: 2767b706cceb48a5bf5440ff912fdea8
I20260810 14:40:57.021332 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.016978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.15.84.60:46403)
0810 14:40:57.017418 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:57.017429 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:57.017853 (+ 424us) server_negotiation.cc:407] Connection header received
0810 14:40:57.017900 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:57.017903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:57.017946 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:57.018022 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:57.018361 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.019062 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.020109 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.020711 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.020805 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.020858 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:57.021071 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 14:40:57.021146 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":300,"thread_start_us":108,"threads_started":1}
I20260810 14:40:57.021410 27067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 14:40:57.021462 27067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2767b706cceb48a5bf5440ff912fdea8)
I20260810 14:40:57.021497 27067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:40:57.021709 26238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee49a0d980e24a6a9d712deafedd3091" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" is_pre_election: true
I20260810 14:40:57.022823 27067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } }
I20260810 14:40:57.023181 27067 leader_election.cc:288] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [CANDIDATE]: Term 2 election: Requested vote from peers 2767b706cceb48a5bf5440ff912fdea8 (127.15.84.62:36385), ee49a0d980e24a6a9d712deafedd3091 (127.15.84.61:36739)
I20260810 14:40:57.023366 26238 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee49a0d980e24a6a9d712deafedd3091 in current term 2: Already voted for candidate 3b9146920cbc4e7a9dd800bb40b11d7d in this term.
I20260810 14:40:57.023521 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.023292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.0.0.1:44410)
0810 14:40:57.023398 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.023452 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 14:40:57.023742 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ee49a0d980e24a6a9d712deafedd3091"
I20260810 14:40:57.023874 25932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:40:57.024170 25898 leader_election.cc:302] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee49a0d980e24a6a9d712deafedd3091; no voters: 2767b706cceb48a5bf5440ff912fdea8, 3b9146920cbc4e7a9dd800bb40b11d7d
I20260810 14:40:57.024883 25932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9146920cbc4e7a9dd800bb40b11d7d in term 2.
I20260810 14:40:57.024987 27066 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:40:57.025295 26189 leader_election.cc:302] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b9146920cbc4e7a9dd800bb40b11d7d, ee49a0d980e24a6a9d712deafedd3091; no voters:
I20260810 14:40:57.025487 27067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:40:57.025715 27067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [term 2 LEADER]: Becoming Leader. State: Replica: 3b9146920cbc4e7a9dd800bb40b11d7d, State: Running, Role: LEADER
I20260810 14:40:57.025796 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.025539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.0.0.1:44412)
0810 14:40:57.025682 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.025735 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260810 14:40:57.025867 26189 leader_election.cc:334] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2767b706cceb48a5bf5440ff912fdea8 (127.15.84.62:36385): Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
I20260810 14:40:57.025997 27067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } }
I20260810 14:40:57.027012 27080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b9146920cbc4e7a9dd800bb40b11d7d. Latest consensus state: current_term: 2 leader_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } } }
I20260810 14:40:57.027103 27080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:57.027379 27082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:57.029667 27082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:57.030031 27082 catalog_manager.cc:1259] Loaded cluster ID: 53b69ac8f06e4caaaecacf0a13e1c527
I20260810 14:40:57.030072 27082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:57.030582 27082 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:57.030956 27082 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: Loaded TSK: 0
I20260810 14:40:57.031674 27082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:40:57.090206 26219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:40:57.093467 26219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:57.096427 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.096182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.0.0.1:44432)
0810 14:40:57.096296 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.096355 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 14:40:57.096781 25932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [term 2 FOLLOWER]: Refusing update from remote peer 3b9146920cbc4e7a9dd800bb40b11d7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 14:40:57.097131 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.096899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.0.0.1:44444)
0810 14:40:57.096999 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.097060 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 14:40:57.097308 26189 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d -> Peer 2767b706cceb48a5bf5440ff912fdea8 (127.15.84.62:36385): Couldn't send request to peer 2767b706cceb48a5bf5440ff912fdea8. Status: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 14:40:57.097342 27067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 14:40:57.098858 27066 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b9146920cbc4e7a9dd800bb40b11d7d. Latest consensus state: current_term: 2 leader_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } } }
I20260810 14:40:57.099025 27066 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:57.100116 27067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } } }
I20260810 14:40:57.100248 27067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:57.100835 27066 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2767b706cceb48a5bf5440ff912fdea8" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36385 } } peers { permanent_uuid: "ee49a0d980e24a6a9d712deafedd3091" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 36739 } attrs { promote: false } } peers { permanent_uuid: "3b9146920cbc4e7a9dd800bb40b11d7d" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 46403 } attrs { promote: false } } }
I20260810 14:40:57.100961 27066 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee49a0d980e24a6a9d712deafedd3091 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:57.111168 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.106926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:33165 (local address 127.0.0.1:35246)
0810 14:40:57.107321 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.107334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:57.107402 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:57.107651 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:57.107653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:57.107671 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:57.107779 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:57.107785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.108692 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.108695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:57.109592 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:57.109601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.110743 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.110746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:57.110836 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.110864 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 14:40:57.110937 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 14:40:57.111006 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":21}
I20260810 14:40:57.111207 27091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.107093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.15.84.1:33165)
0810 14:40:57.107402 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:57.107406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:57.107421 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:40:57.107477 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:57.107481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:57.107532 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:57.107620 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:57.107918 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.108590 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.109730 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.110519 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.110733 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.110806 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:57.110962 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 14:40:57.111052 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":101,"threads_started":1}
I20260810 14:40:57.111464 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.106614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:35725 (local address 127.0.0.1:33704)
0810 14:40:57.106861 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.106877 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:57.106956 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:57.108084 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:57.108087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:57.108106 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:57.108191 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:57.108196 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.109174 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.109178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:57.110264 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:57.110274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.111114 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.111118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:57.111203 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.111229 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:40:57.111280 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 14:40:57.111350 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":7}
I20260810 14:40:57.111766 27095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.107424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:46843 (local address 127.0.0.1:33026)
0810 14:40:57.107893 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.107905 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:57.107988 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:57.108413 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:57.108416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:57.108433 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:57.108546 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:57.108551 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.109593 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.109596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:57.110529 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:57.110539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.111365 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.111368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:57.111474 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.111500 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:40:57.111554 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 14:40:57.111627 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":102,"threads_started":1}
I20260810 14:40:57.111940 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.107116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.15.84.2:35725)
0810 14:40:57.107860 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:57.107865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:57.107878 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:40:57.107929 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:57.107933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:57.107978 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:57.108062 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:57.108301 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.109028 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.110417 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.110990 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.111628 (+ 638us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.111706 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:57.111742 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 14:40:57.111815 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":95,"threads_started":1}
I20260810 14:40:57.112067 26404 tablet_service.cc:1467] Processing CreateTablet for tablet b0727031ce1844398411a5817cfac756 (DEFAULT_TABLE table=test-table [id=b68f09d16366454bb37e2090673993a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:57.112260 27097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.107812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.15.84.3:46843)
0810 14:40:57.108187 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:57.108192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:57.108205 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:40:57.108249 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:57.108252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:57.108295 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:57.108645 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:57.108752 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.109476 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.110640 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.111232 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.111953 (+ 721us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.112023 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:57.112063 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 14:40:57.112134 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":93,"threads_started":1}
I20260810 14:40:57.112416 26404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0727031ce1844398411a5817cfac756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:57.113673 26626 tablet_service.cc:1467] Processing CreateTablet for tablet b0727031ce1844398411a5817cfac756 (DEFAULT_TABLE table=test-table [id=b68f09d16366454bb37e2090673993a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:57.113967 26626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0727031ce1844398411a5817cfac756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:57.114928 27098 tablet_bootstrap.cc:492] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d: Bootstrap starting.
I20260810 14:40:57.115783 27098 tablet_bootstrap.cc:654] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:57.116144 27099 tablet_bootstrap.cc:492] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389: Bootstrap starting.
I20260810 14:40:57.116520 26850 tablet_service.cc:1467] Processing CreateTablet for tablet b0727031ce1844398411a5817cfac756 (DEFAULT_TABLE table=test-table [id=b68f09d16366454bb37e2090673993a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:57.116830 26850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0727031ce1844398411a5817cfac756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:57.117055 27099 tablet_bootstrap.cc:654] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:57.117571 27098 tablet_bootstrap.cc:492] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d: No bootstrap required, opened a new log
I20260810 14:40:57.117974 27098 ts_tablet_manager.cc:1397] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:40:57.118362 27099 tablet_bootstrap.cc:492] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389: No bootstrap required, opened a new log
I20260810 14:40:57.118449 27099 ts_tablet_manager.cc:1397] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 14:40:57.118450 27098 raft_consensus.cc:348] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } }
I20260810 14:40:57.118544 27098 raft_consensus.cc:374] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:57.118563 27098 raft_consensus.cc:729] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6e95f155e54ee58454b5767f2ba60d, State: Initialized, Role: FOLLOWER
I20260810 14:40:57.118685 27098 consensus_queue.cc:260] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } }
I20260810 14:40:57.118870 27099 raft_consensus.cc:348] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } }
I20260810 14:40:57.118983 27099 raft_consensus.cc:374] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:57.119016 27099 raft_consensus.cc:729] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2561cc3377241c0a83657629ac58389, State: Initialized, Role: FOLLOWER
I20260810 14:40:57.119010 27098 ts_tablet_manager.cc:1428] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:57.119143 27099 consensus_queue.cc:260] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } }
I20260810 14:40:57.119437 27107 tablet_bootstrap.cc:492] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff: Bootstrap starting.
I20260810 14:40:57.119449 27099 ts_tablet_manager.cc:1428] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:40:57.121616 27107 tablet_bootstrap.cc:654] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:57.122571 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.119908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.2:41671)
0810 14:40:57.122393 (+ 2485us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.122461 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2404,"thread_start_us":94,"threads_started":1}
I20260810 14:40:57.123165 27107 tablet_bootstrap.cc:492] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff: No bootstrap required, opened a new log
I20260810 14:40:57.123252 27107 ts_tablet_manager.cc:1397] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 14:40:57.123654 27107 raft_consensus.cc:348] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } }
I20260810 14:40:57.123761 27107 raft_consensus.cc:374] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:57.123782 27107 raft_consensus.cc:729] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9327ffe3276c48a49fa54211a6e32fff, State: Initialized, Role: FOLLOWER
I20260810 14:40:57.123888 27107 consensus_queue.cc:260] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } }
I20260810 14:40:57.124145 27107 ts_tablet_manager.cc:1428] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:57.124444 26742 heartbeater.cc:503] Master 127.15.84.60:46403 was elected leader, sending a full tablet report...
I20260810 14:40:57.124953 27115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.124513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.3:44589)
0810 14:40:57.124789 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.124853 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":110,"threads_started":1}
I20260810 14:40:57.125530 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.119327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.1:41731)
0810 14:40:57.125366 (+ 6039us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.125428 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5953,"thread_start_us":103,"threads_started":1}
I20260810 14:40:57.125787 27115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.125389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.3:47417)
0810 14:40:57.125669 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.125717 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":94,"threads_started":1}
I20260810 14:40:57.126490 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.126014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.1:50185)
0810 14:40:57.126270 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.126392 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20260810 14:40:57.126848 26959 heartbeater.cc:503] Master 127.15.84.60:46403 was elected leader, sending a full tablet report...
I20260810 14:40:57.127156 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.126912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.2:33403)
0810 14:40:57.127023 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.127083 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 14:40:57.224675 27114 raft_consensus.cc:482] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:57.224867 27114 raft_consensus.cc:504] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } }
I20260810 14:40:57.226100 27114 leader_election.cc:288] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2561cc3377241c0a83657629ac58389 (127.15.84.2:35725), aa6e95f155e54ee58454b5767f2ba60d (127.15.84.1:33165)
I20260810 14:40:57.230566 27091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.226497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:34361 (local address 127.15.84.1:33165)
0810 14:40:57.226621 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:57.226626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:57.226830 (+ 204us) server_negotiation.cc:407] Connection header received
0810 14:40:57.226921 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:57.226923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:57.226973 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:57.227047 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:57.227335 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.228166 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.229569 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.229986 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.230084 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.230157 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:57.230374 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 14:40:57.230459 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:40:57.230587 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.226369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:33165 (local address 127.15.84.3:34361)
0810 14:40:57.226744 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.226757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:57.226849 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:57.227060 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:57.227062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:57.227075 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:57.227206 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:57.227214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.228338 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.228342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:57.229401 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:57.229414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.230140 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.230145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:57.230234 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.230259 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:40:57.230338 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 14:40:57.230421 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":137}
I20260810 14:40:57.231060 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0727031ce1844398411a5817cfac756" candidate_uuid: "9327ffe3276c48a49fa54211a6e32fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6e95f155e54ee58454b5767f2ba60d" is_pre_election: true
I20260810 14:40:57.231225 26427 raft_consensus.cc:2454] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9327ffe3276c48a49fa54211a6e32fff in term 0.
I20260810 14:40:57.231294 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.226611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:35561 (local address 127.15.84.2:35725)
0810 14:40:57.226756 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:57.226761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:57.226778 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:40:57.226843 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:57.226846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:57.226904 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:57.227001 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:57.227335 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.228166 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.230235 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.230813 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.230914 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.230976 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:57.231111 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 14:40:57.231184 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:40:57.231297 27115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.226369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:35725 (local address 127.15.84.3:35561)
0810 14:40:57.226599 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.226616 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:57.226731 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:57.227009 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:57.227011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:57.227028 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:57.227206 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:57.227213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.228291 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.228295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:57.229403 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:57.229415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.230912 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.230915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:57.231001 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.231023 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:40:57.231088 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 14:40:57.231171 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":69,"spinlock_wait_cycles":1792}
I20260810 14:40:57.231657 26783 leader_election.cc:302] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9327ffe3276c48a49fa54211a6e32fff, aa6e95f155e54ee58454b5767f2ba60d; no voters:
I20260810 14:40:57.231974 27114 raft_consensus.cc:2781] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:40:57.231946 26661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0727031ce1844398411a5817cfac756" candidate_uuid: "9327ffe3276c48a49fa54211a6e32fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2561cc3377241c0a83657629ac58389" is_pre_election: true
I20260810 14:40:57.232039 27114 raft_consensus.cc:482] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:40:57.232075 27114 raft_consensus.cc:3037] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:57.232210 26661 raft_consensus.cc:2454] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9327ffe3276c48a49fa54211a6e32fff in term 0.
I20260810 14:40:57.233290 27114 raft_consensus.cc:504] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } }
I20260810 14:40:57.233625 27114 leader_election.cc:288] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [CANDIDATE]: Term 1 election: Requested vote from peers f2561cc3377241c0a83657629ac58389 (127.15.84.2:35725), aa6e95f155e54ee58454b5767f2ba60d (127.15.84.1:33165)
I20260810 14:40:57.233953 26661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0727031ce1844398411a5817cfac756" candidate_uuid: "9327ffe3276c48a49fa54211a6e32fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2561cc3377241c0a83657629ac58389"
I20260810 14:40:57.234061 26661 raft_consensus.cc:3037] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:57.234023 26427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0727031ce1844398411a5817cfac756" candidate_uuid: "9327ffe3276c48a49fa54211a6e32fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6e95f155e54ee58454b5767f2ba60d"
I20260810 14:40:57.234158 26427 raft_consensus.cc:3037] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:57.235136 26661 raft_consensus.cc:2454] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9327ffe3276c48a49fa54211a6e32fff in term 1.
I20260810 14:40:57.235260 26427 raft_consensus.cc:2454] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9327ffe3276c48a49fa54211a6e32fff in term 1.
I20260810 14:40:57.235450 26779 leader_election.cc:302] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9327ffe3276c48a49fa54211a6e32fff, f2561cc3377241c0a83657629ac58389; no voters:
I20260810 14:40:57.235724 27114 raft_consensus.cc:2781] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:57.235966 27114 raft_consensus.cc:686] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [term 1 LEADER]: Becoming Leader. State: Replica: 9327ffe3276c48a49fa54211a6e32fff, State: Running, Role: LEADER
I20260810 14:40:57.236421 27114 consensus_queue.cc:237] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } }
I20260810 14:40:57.236730 27115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.236421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.3:57197)
0810 14:40:57.236546 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.236631 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 14:40:57.239194 26219 catalog_manager.cc:5495] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff reported cstate change: term changed from 0 to 1, leader changed from <none> to 9327ffe3276c48a49fa54211a6e32fff (127.15.84.3). New cstate: current_term: 1 leader_uuid: "9327ffe3276c48a49fa54211a6e32fff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9327ffe3276c48a49fa54211a6e32fff" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46843 } health_report { overall_health: HEALTHY } } }
W20260810 14:40:57.267120 26963 tablet.cc:2369] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:40:57.267287 27097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.263406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.15.84.3:46843)
0810 14:40:57.263524 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:57.263528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:57.263541 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:40:57.263701 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:57.263703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:57.263761 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:57.263855 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:57.264097 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.264630 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.265862 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:57.266085 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:57.266174 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:57.266415 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:57.266893 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:57.266983 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:57.267082 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 14:40:57.267165 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 14:40:57.270299 26427 raft_consensus.cc:1261] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d [term 1 FOLLOWER]: Refusing update from remote peer 9327ffe3276c48a49fa54211a6e32fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:57.270669 26661 raft_consensus.cc:1261] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389 [term 1 FOLLOWER]: Refusing update from remote peer 9327ffe3276c48a49fa54211a6e32fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:57.270843 27114 consensus_queue.cc:1035] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa6e95f155e54ee58454b5767f2ba60d" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 33165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:57.271279 27122 consensus_queue.cc:1035] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2561cc3377241c0a83657629ac58389" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:57.272141 26494 heartbeater.cc:503] Master 127.15.84.60:46403 was elected leader, sending a full tablet report...
I20260810 14:40:57.272599 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.272333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.1:41789)
0810 14:40:57.272438 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.272504 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 14:40:57.274472 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.274213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.2:56319)
0810 14:40:57.274332 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.274394 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 14:40:57.277127 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.276766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.1:47559)
0810 14:40:57.276977 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.277040 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260810 14:40:57.277212 27115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.276767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.3:57831)
0810 14:40:57.277071 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.277120 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232}
I20260810 14:40:57.278138 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.277452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36385 (local address 127.15.84.2:43051)
0810 14:40:57.278006 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.278065 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479}
I20260810 14:40:57.285950 26219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:57.286147 26219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:57.290745 26219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b9146920cbc4e7a9dd800bb40b11d7d: Sending DeleteTablet for 3 replicas of tablet b0727031ce1844398411a5817cfac756
I20260810 14:40:57.291433 26626 tablet_service.cc:1514] Processing DeleteTablet for tablet b0727031ce1844398411a5817cfac756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:57 UTC) from {username='oryx', principal='oryx/127.15.84.60@KRBTEST.COM'} at 127.0.0.1:33704
I20260810 14:40:57.291596 26404 tablet_service.cc:1514] Processing DeleteTablet for tablet b0727031ce1844398411a5817cfac756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:57 UTC) from {username='oryx', principal='oryx/127.15.84.60@KRBTEST.COM'} at 127.0.0.1:35246
I20260810 14:40:57.291769 26850 tablet_service.cc:1514] Processing DeleteTablet for tablet b0727031ce1844398411a5817cfac756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:57 UTC) from {username='oryx', principal='oryx/127.15.84.60@KRBTEST.COM'} at 127.0.0.1:33026
I20260810 14:40:57.292032 27145 tablet_replica.cc:330] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d: stopping tablet replica
I20260810 14:40:57.292171 27145 raft_consensus.cc:2229] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:57.292323 27145 raft_consensus.cc:2258] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:57.292424 27144 tablet_replica.cc:330] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389: stopping tablet replica
I20260810 14:40:57.292734 27144 raft_consensus.cc:2229] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:57.292896 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 26140
I20260810 14:40:57.293059 27145 ts_tablet_manager.cc:1905] T b0727031ce1844398411a5817cfac756 P aa6e95f155e54ee58454b5767f2ba60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:57.296044 27144 raft_consensus.cc:2258] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:57.296875 27144 ts_tablet_manager.cc:1905] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:57.299741 27144 ts_tablet_manager.cc:1918] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:57.299825 27144 log.cc:1196] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/b0727031ce1844398411a5817cfac756
I20260810 14:40:57.300107 27144 ts_tablet_manager.cc:1939] T b0727031ce1844398411a5817cfac756 P f2561cc3377241c0a83657629ac58389: Deleting consensus metadata
I20260810 14:40:57.300889 26188 catalog_manager.cc:4841] TS f2561cc3377241c0a83657629ac58389 (127.15.84.2:35725): tablet b0727031ce1844398411a5817cfac756 (table test-table [id=b68f09d16366454bb37e2090673993a2]) successfully deleted
I20260810 14:40:57.305433 27146 tablet_replica.cc:330] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff: stopping tablet replica
I20260810 14:40:57.305605 27146 raft_consensus.cc:2229] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:40:57.305987 27146 raft_consensus.cc:2258] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:57.306775 27146 ts_tablet_manager.cc:1905] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:57.309629 27146 ts_tablet_manager.cc:1918] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:57.309733 27146 log.cc:1196] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/b0727031ce1844398411a5817cfac756
I20260810 14:40:57.310009 27146 ts_tablet_manager.cc:1939] T b0727031ce1844398411a5817cfac756 P 9327ffe3276c48a49fa54211a6e32fff: Deleting consensus metadata
I20260810 14:40:57.310948 26188 catalog_manager.cc:4841] TS 9327ffe3276c48a49fa54211a6e32fff (127.15.84.3:46843): tablet b0727031ce1844398411a5817cfac756 (table test-table [id=b68f09d16366454bb37e2090673993a2]) successfully deleted
I20260810 14:40:57.312412 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 26510
I20260810 14:40:57.312675 27095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:57.312353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:33165 (local address 127.0.0.1:35258)
0810 14:40:57.312513 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:57.312579 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:33165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 14:40:57.322204 26191 catalog_manager.cc:4584] TS aa6e95f155e54ee58454b5767f2ba60d (127.15.84.1:33165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0727031ce1844398411a5817cfac756: Network error: Client connection negotiation failed: client connection to 127.15.84.1:33165: connect: Connection refused (error 111)
I20260810 14:40:57.332597 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 26755
I20260810 14:40:57.347311 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 25701
I20260810 14:40:57.367375 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 25878
2026-08-10T14:40:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5785 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:57 dist-test-slave-rmhg krb5kdc[27156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:57 dist-test-slave-rmhg krb5kdc[27156](info): set up 2 sockets
Aug 10 14:40:57 dist-test-slave-rmhg krb5kdc[27156](info): commencing operation
krb5kdc: starting...
W20260810 14:40:58.522562 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:40:58 dist-test-slave-rmhg krb5kdc[27156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.62.keytab.
Entry for principal oryx/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.62.keytab.
I20260810 14:40:58.652093 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:37551
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.62.keytab
--principal=oryx/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:37551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:58.770396 27171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:58.770730 27171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:58.770793 27171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:58.775045 27171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:58.775135 27171 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 14:40:58.775159 27171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:58.775183 27171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:58.775204 27171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:58.775225 27171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:58.781266 27171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:37551
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.62.keytab
--principal=oryx/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:37551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:58.782512 27171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:58.783636 27171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:58.793107 27177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:58.793083 27178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:58.793079 27182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:58.793167 27171 server_base.cc:1034] running on GCE node
I20260810 14:40:58.794049 27171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:58.795311 27171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:58.796473 27171 hybrid_clock.cc:648] HybridClock initialized: now 1786372858796448 us; error 45 us; skew 500 ppm
Aug 10 14:40:58 dist-test-slave-rmhg krb5kdc[27156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372858, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:58.799432 27171 init.cc:377] Logged in from keytab as oryx/127.15.84.62@KRBTEST.COM (short username oryx)
I20260810 14:40:58.800453 27171 webserver.cc:460] Webserver started at http://127.15.84.62:34301/ using document root <none> and password file <none>
I20260810 14:40:58.801062 27171 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:58.801131 27171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:58.801378 27171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:58.803700 27171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "dc6e78ae3e3a427cb60b967c0f5c9dde"
format_stamp: "Formatted at 2026-08-10 14:40:58 on dist-test-slave-rmhg"
server_key: "3102e2505df02d391fa641bae4087a85"
server_key_iv: "6dc5c894e9398a13c188014af5d7e475"
server_key_version: "encryptionkey@0"
I20260810 14:40:58.804325 27171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "dc6e78ae3e3a427cb60b967c0f5c9dde"
format_stamp: "Formatted at 2026-08-10 14:40:58 on dist-test-slave-rmhg"
server_key: "3102e2505df02d391fa641bae4087a85"
server_key_iv: "6dc5c894e9398a13c188014af5d7e475"
server_key_version: "encryptionkey@0"
I20260810 14:40:58.808372 27171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 14:40:58.811048 27193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:58.812397 27171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:40:58.812536 27171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "dc6e78ae3e3a427cb60b967c0f5c9dde"
format_stamp: "Formatted at 2026-08-10 14:40:58 on dist-test-slave-rmhg"
server_key: "3102e2505df02d391fa641bae4087a85"
server_key_iv: "6dc5c894e9398a13c188014af5d7e475"
server_key_version: "encryptionkey@0"
I20260810 14:40:58.812662 27171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:58.827471 27171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:58.830721 27171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:58.830943 27171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:58.839331 27171 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:37551
I20260810 14:40:58.839335 27292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:37551 every 8 connection(s)
I20260810 14:40:58.840472 27171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:58.843560 27293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:58.849704 27293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde: Bootstrap starting.
I20260810 14:40:58.850121 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 27171
I20260810 14:40:58.850248 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:40:58.850497 15696 external_mini_cluster.cc:1428] Setting key 1b28c87a77da0713358c6b90ce2250af
I20260810 14:40:58.852406 27293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:58.853212 27293 log.cc:824] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:58.855527 27293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde: No bootstrap required, opened a new log
I20260810 14:40:58.859632 27293 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6e78ae3e3a427cb60b967c0f5c9dde" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37551 } }
Aug 10 14:40:58 dist-test-slave-rmhg krb5kdc[27156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
I20260810 14:40:58.859903 27293 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:58.859973 27293 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6e78ae3e3a427cb60b967c0f5c9dde, State: Initialized, Role: FOLLOWER
I20260810 14:40:58.860445 27293 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6e78ae3e3a427cb60b967c0f5c9dde" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37551 } }
I20260810 14:40:58.860611 27293 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:58.860703 27293 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:58.860816 27293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:58.862048 27293 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6e78ae3e3a427cb60b967c0f5c9dde" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37551 } }
I20260810 14:40:58.862450 27293 leader_election.cc:302] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc6e78ae3e3a427cb60b967c0f5c9dde; no voters:
I20260810 14:40:58.862824 27293 leader_election.cc:288] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:58.862943 27306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:58.863188 27306 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [term 1 LEADER]: Becoming Leader. State: Replica: dc6e78ae3e3a427cb60b967c0f5c9dde, State: Running, Role: LEADER
I20260810 14:40:58.863494 27306 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6e78ae3e3a427cb60b967c0f5c9dde" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37551 } }
I20260810 14:40:58.864265 27293 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:58.866331 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:58.851828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.15.84.62:37551)
0810 14:40:58.852297 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:58.852314 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:58.852350 (+ 36us) server_negotiation.cc:407] Connection header received
0810 14:40:58.853286 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:58.853315 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:58.853698 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:58.854031 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:58.854716 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:58.856200 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:58.857080 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:58.857881 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:58.858052 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:58.860509 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:58.860516 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:58.860535 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:58.860582 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:58.862542 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:58.863079 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:58.863085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:58.863092 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:58.863237 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:58.863637 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:58.863640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:58.863642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:58.864168 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:58.864363 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:58.865076 (+ 713us) server_negotiation.cc:299] Negotiation successful
0810 14:40:58.865435 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":150,"threads_started":1}
I20260810 14:40:58.867090 27308 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc6e78ae3e3a427cb60b967c0f5c9dde. Latest consensus state: current_term: 1 leader_uuid: "dc6e78ae3e3a427cb60b967c0f5c9dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6e78ae3e3a427cb60b967c0f5c9dde" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37551 } } }
I20260810 14:40:58.867072 27307 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc6e78ae3e3a427cb60b967c0f5c9dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6e78ae3e3a427cb60b967c0f5c9dde" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37551 } } }
I20260810 14:40:58.867395 27307 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:58.867395 27308 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde [sys.catalog]: This master's current role is: LEADER
W20260810 14:40:58.870488 27327 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:40:58.870568 27327 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:40:58.870642 27328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:58.871498 27328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:58.877831 27328 catalog_manager.cc:1347] Generated new cluster ID: 2014f09308884370ad1655d99e7f1ec8
I20260810 14:40:58.877918 27328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:58.894251 27328 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:58.895301 27328 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:58.901475 27328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde: Generated new TSK 0
I20260810 14:40:58.902207 27328 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.1.keytab.
Entry for principal oryx/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.1.keytab.
I20260810 14:40:58.936040 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:37551
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.1.keytab
--principal=oryx/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:59.053272 27337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:59.053542 27337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:59.053588 27337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:59.057612 27337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:59.057675 27337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:59.057715 27337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:40:59.057752 27337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:40:59.063091 27337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.1.keytab
--principal=oryx/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.84.62:37551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:59.064123 27337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:59.065208 27337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:59.075099 27347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:59.075208 27337 server_base.cc:1034] running on GCE node
W20260810 14:40:59.075129 27345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:59.075155 27344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:59.075757 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:59.076399 27337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:59.077557 27337 hybrid_clock.cc:648] HybridClock initialized: now 1786372859077529 us; error 39 us; skew 500 ppm
Aug 10 14:40:59 dist-test-slave-rmhg krb5kdc[27156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372859, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:59.080420 27337 init.cc:377] Logged in from keytab as oryx/127.15.84.1@KRBTEST.COM (short username oryx)
I20260810 14:40:59.081482 27337 webserver.cc:460] Webserver started at http://127.15.84.1:41007/ using document root <none> and password file <none>
I20260810 14:40:59.082073 27337 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:59.082145 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:59.082362 27337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:59.084619 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "f48dd5c98ead413e881a9187a8678331"
format_stamp: "Formatted at 2026-08-10 14:40:59 on dist-test-slave-rmhg"
server_key: "a1d044cca700118f766abd13eff7b76b"
server_key_iv: "4637739a81ec5f31ff87f1bd77a68247"
server_key_version: "encryptionkey@0"
I20260810 14:40:59.085283 27337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "f48dd5c98ead413e881a9187a8678331"
format_stamp: "Formatted at 2026-08-10 14:40:59 on dist-test-slave-rmhg"
server_key: "a1d044cca700118f766abd13eff7b76b"
server_key_iv: "4637739a81ec5f31ff87f1bd77a68247"
server_key_version: "encryptionkey@0"
I20260810 14:40:59.089293 27337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:40:59.091931 27360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:59.093384 27337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:59.093573 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "f48dd5c98ead413e881a9187a8678331"
format_stamp: "Formatted at 2026-08-10 14:40:59 on dist-test-slave-rmhg"
server_key: "a1d044cca700118f766abd13eff7b76b"
server_key_iv: "4637739a81ec5f31ff87f1bd77a68247"
server_key_version: "encryptionkey@0"
I20260810 14:40:59.093709 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:59.113273 27337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:59.116637 27337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:59.116845 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:59.118595 27337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:59.118656 27337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:59.118716 27337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:59.118746 27337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:40:59 dist-test-slave-rmhg krb5kdc[27156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372859, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
I20260810 14:40:59.130183 27337 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:43271
I20260810 14:40:59.130286 27549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:43271 every 8 connection(s)
I20260810 14:40:59.131356 27337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:40:59.132925 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.119455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:35193 (local address 127.15.84.62:37551)
0810 14:40:59.119624 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:59.119630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:59.120614 (+ 984us) server_negotiation.cc:407] Connection header received
0810 14:40:59.121811 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:59.121816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:59.121882 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:59.121943 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:59.123551 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.124170 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.125469 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.125758 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.125858 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.128920 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:59.128925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:59.128928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:59.128965 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:59.130662 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:59.131192 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:59.131196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:59.131198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:59.131304 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:59.131662 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:59.131665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:59.131667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:59.131936 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:59.132053 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:59.132642 (+ 589us) server_negotiation.cc:299] Negotiation successful
0810 14:40:59.132781 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:40:59.133694 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 27337
I20260810 14:40:59.133800 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:40:59.133817 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.119886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37551 (local address 127.15.84.1:35193)
0810 14:40:59.120417 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:59.120473 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:59.121522 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:59.122389 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:59.122396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:59.122769 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:59.123389 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.123411 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.124305 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.124309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.125341 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.125352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.125938 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.125942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.126183 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.126848 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:59.126870 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:59.128721 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:59.130788 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:59.130793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:59.130805 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:59.131077 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:59.131396 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:59.131398 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:59.131399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:59.131565 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:59.132114 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:59.132121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:59.132396 (+ 275us) client_negotiation.cc:769] Sending connection context
0810 14:40:59.132568 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 14:40:59.132871 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":125,"threads_started":1}
I20260810 14:40:59.134116 15696 external_mini_cluster.cc:1428] Setting key 8bfa6ee68d2a3ba55c409739c5dd9d41
I20260810 14:40:59.135776 27552 heartbeater.cc:347] Connected to a master server at 127.15.84.62:37551
I20260810 14:40:59.135933 27552 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:59.136502 27552 heartbeater.cc:511] Master 127.15.84.62:37551 requested a full tablet report, sending...
I20260810 14:40:59.138137 27214 ts_manager.cc:194] Registered new tserver with Master: f48dd5c98ead413e881a9187a8678331 (127.15.84.1:43271)
I20260810 14:40:59.139344 27214 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:35193
I20260810 14:40:59.146541 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.137939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:46865 (local address 127.15.84.62:37551)
0810 14:40:59.139946 (+ 2007us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:59.139950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:59.139966 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:40:59.140009 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:59.140012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:59.140053 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:59.140137 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:59.140633 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.141196 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.142013 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.142237 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.142328 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.143497 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:59.143500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:59.143502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:59.143558 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:59.144918 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:59.145380 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:59.145384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:59.145386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:59.145467 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:59.145726 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:59.145729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:59.145731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:59.145908 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:59.145995 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:59.146260 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 14:40:59.146369 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1915}
I20260810 14:40:59.146541 27562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.137862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37551 (local address 127.15.84.1:46865)
0810 14:40:59.138186 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:59.138204 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:59.138300 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:59.140179 (+ 1879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:59.140183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:59.140390 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:59.140507 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.140514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.141322 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.141325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.141898 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.141907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.142344 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.142347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.142446 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.143006 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:59.143017 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:59.143357 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:59.145036 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:59.145039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:59.145041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:59.145288 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:59.145554 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:59.145557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:59.145559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:59.145646 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:59.146000 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:59.146003 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:59.146108 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 14:40:59.146198 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 14:40:59.146359 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":136,"threads_started":1}
WARNING: no policy specified for oryx/127.15.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.2.keytab.
Entry for principal oryx/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.2.keytab.
I20260810 14:40:59.166214 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:37551
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.2.keytab
--principal=oryx/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:59.285900 27566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:59.286298 27566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:59.286374 27566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:59.290506 27566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:59.290578 27566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:59.290632 27566 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:40:59.290697 27566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:40:59.296260 27566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.2.keytab
--principal=oryx/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.84.62:37551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:59.297500 27566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:59.298583 27566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:59.308238 27574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:59.308346 27578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:59.308408 27566 server_base.cc:1034] running on GCE node
W20260810 14:40:59.308238 27581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:59.308899 27566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:59.309603 27566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:59.310774 27566 hybrid_clock.cc:648] HybridClock initialized: now 1786372859310757 us; error 37 us; skew 500 ppm
Aug 10 14:40:59 dist-test-slave-rmhg krb5kdc[27156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372859, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:59.313633 27566 init.cc:377] Logged in from keytab as oryx/127.15.84.2@KRBTEST.COM (short username oryx)
I20260810 14:40:59.314587 27566 webserver.cc:460] Webserver started at http://127.15.84.2:39807/ using document root <none> and password file <none>
I20260810 14:40:59.315150 27566 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:59.315224 27566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:59.315433 27566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:59.317703 27566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "f4d041883b7f4dfc8653f1bca560fd9a"
format_stamp: "Formatted at 2026-08-10 14:40:59 on dist-test-slave-rmhg"
server_key: "185ea7d126c5b150afe4bca1114b4790"
server_key_iv: "05f05e3d3ae7fef79e7c7d1ec5b08c09"
server_key_version: "encryptionkey@0"
I20260810 14:40:59.318293 27566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "f4d041883b7f4dfc8653f1bca560fd9a"
format_stamp: "Formatted at 2026-08-10 14:40:59 on dist-test-slave-rmhg"
server_key: "185ea7d126c5b150afe4bca1114b4790"
server_key_iv: "05f05e3d3ae7fef79e7c7d1ec5b08c09"
server_key_version: "encryptionkey@0"
I20260810 14:40:59.322315 27566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:40:59.324985 27591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:59.326318 27566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:59.326449 27566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "f4d041883b7f4dfc8653f1bca560fd9a"
format_stamp: "Formatted at 2026-08-10 14:40:59 on dist-test-slave-rmhg"
server_key: "185ea7d126c5b150afe4bca1114b4790"
server_key_iv: "05f05e3d3ae7fef79e7c7d1ec5b08c09"
server_key_version: "encryptionkey@0"
I20260810 14:40:59.326562 27566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:59.342262 27566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:59.345803 27566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:59.345997 27566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:59.347584 27566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:59.347641 27566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:59.347692 27566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:59.347708 27566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:40:59 dist-test-slave-rmhg krb5kdc[27156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372859, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
I20260810 14:40:59.358594 27566 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:35349
I20260810 14:40:59.359138 27778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:35349 every 8 connection(s)
I20260810 14:40:59.359833 27566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:40:59.361760 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.348429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:35833 (local address 127.15.84.62:37551)
0810 14:40:59.348586 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:59.348591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:59.349564 (+ 973us) server_negotiation.cc:407] Connection header received
0810 14:40:59.350614 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:59.350618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:59.350681 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:59.350745 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:59.352377 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.353076 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.354395 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.354632 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.354740 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.357699 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:59.357702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:59.357704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:59.357734 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:59.359447 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:59.360097 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:59.360101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:59.360102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:59.360216 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:59.360626 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:59.360630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:59.360632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:59.360860 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:59.360956 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:59.361509 (+ 553us) server_negotiation.cc:299] Negotiation successful
0810 14:40:59.361627 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:40:59.362581 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.348861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37551 (local address 127.15.84.2:35833)
0810 14:40:59.349386 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:59.349439 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:59.350351 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:59.351205 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:59.351212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:59.351566 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:59.352222 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.352244 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.353234 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.353237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.354276 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.354289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.354813 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.354817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.355020 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.355707 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:59.355725 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:59.357478 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:59.359578 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:59.359585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:59.359602 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:59.359982 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:59.360321 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:59.360325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:59.360327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:59.360509 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:59.360960 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:59.360967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:59.361283 (+ 316us) client_negotiation.cc:769] Sending connection context
0810 14:40:59.361449 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 14:40:59.361751 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":120,"threads_started":1}
I20260810 14:40:59.363811 27779 heartbeater.cc:347] Connected to a master server at 127.15.84.62:37551
I20260810 14:40:59.363969 27779 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:59.364485 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 27566
I20260810 14:40:59.364601 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:40:59.364655 27779 heartbeater.cc:511] Master 127.15.84.62:37551 requested a full tablet report, sending...
I20260810 14:40:59.364977 15696 external_mini_cluster.cc:1428] Setting key 32748dfb0cef9b7a85ce968b3b616dba
I20260810 14:40:59.365972 27211 ts_manager.cc:194] Registered new tserver with Master: f4d041883b7f4dfc8653f1bca560fd9a (127.15.84.2:35349)
I20260810 14:40:59.366771 27211 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:35833
I20260810 14:40:59.373883 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.366472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:42495 (local address 127.15.84.62:37551)
0810 14:40:59.366594 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:59.366599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:59.366854 (+ 255us) server_negotiation.cc:407] Connection header received
0810 14:40:59.366950 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:59.366953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:59.366995 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:59.367081 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:59.367661 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.368365 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.369402 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.369709 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.369809 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.371006 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:59.371009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:59.371010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:59.371032 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:59.372263 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:59.372650 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:59.372654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:59.372656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:59.372735 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:59.372983 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:59.372986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:59.372988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:59.373271 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:59.373360 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:59.373613 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 14:40:59.373724 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:40:59.373903 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.366411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37551 (local address 127.15.84.2:42495)
0810 14:40:59.366745 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:59.366760 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:59.366870 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:59.367207 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:59.367210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:59.367435 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:59.367551 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.367557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.368509 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.368513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.369279 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.369291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.369829 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.369833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.369931 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.370423 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:59.370437 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:59.370877 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:59.372369 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:59.372372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:59.372374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:59.372563 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:59.372817 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:59.372820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:59.372823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:59.372903 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:59.373361 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:59.373364 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:59.373460 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 14:40:59.373549 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 14:40:59.373713 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for oryx/127.15.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.3.keytab.
Entry for principal oryx/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.3.keytab.
I20260810 14:40:59.396514 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:37551
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.3.keytab
--principal=oryx/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:59.519402 27791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:59.519735 27791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:59.519784 27791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:59.524053 27791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:59.524137 27791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:59.524179 27791 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:40:59.524212 27791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:40:59.530385 27791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.3.keytab
--principal=oryx/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.84.62:37551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:59.531631 27791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:59.532672 27791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:59.542946 27803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:59.542919 27802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:59.542909 27806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:59.543201 27791 server_base.cc:1034] running on GCE node
I20260810 14:40:59.543697 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:59.544370 27791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:59.545591 27791 hybrid_clock.cc:648] HybridClock initialized: now 1786372859545556 us; error 47 us; skew 500 ppm
Aug 10 14:40:59 dist-test-slave-rmhg krb5kdc[27156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372859, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:59.548472 27791 init.cc:377] Logged in from keytab as oryx/127.15.84.3@KRBTEST.COM (short username oryx)
I20260810 14:40:59.549538 27791 webserver.cc:460] Webserver started at http://127.15.84.3:38641/ using document root <none> and password file <none>
I20260810 14:40:59.550133 27791 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:59.550205 27791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:59.550422 27791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:59.552603 27791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "bd476e64d6c8463eb74768b9f6b7822f"
format_stamp: "Formatted at 2026-08-10 14:40:59 on dist-test-slave-rmhg"
server_key: "c858cad266151233ed8e47e0c6787495"
server_key_iv: "d6f2534c2120cfd0dc5b37ba0fdb9276"
server_key_version: "encryptionkey@0"
I20260810 14:40:59.553237 27791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "bd476e64d6c8463eb74768b9f6b7822f"
format_stamp: "Formatted at 2026-08-10 14:40:59 on dist-test-slave-rmhg"
server_key: "c858cad266151233ed8e47e0c6787495"
server_key_iv: "d6f2534c2120cfd0dc5b37ba0fdb9276"
server_key_version: "encryptionkey@0"
I20260810 14:40:59.557380 27791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:40:59.560372 27816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:59.561838 27791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:59.561978 27791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "bd476e64d6c8463eb74768b9f6b7822f"
format_stamp: "Formatted at 2026-08-10 14:40:59 on dist-test-slave-rmhg"
server_key: "c858cad266151233ed8e47e0c6787495"
server_key_iv: "d6f2534c2120cfd0dc5b37ba0fdb9276"
server_key_version: "encryptionkey@0"
I20260810 14:40:59.562100 27791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:59.579171 27791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:59.582727 27791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:59.582935 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:59.584734 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:40:59.584810 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:59.584864 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:40:59.584893 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:40:59 dist-test-slave-rmhg krb5kdc[27156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372859, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
I20260810 14:40:59.595963 27791 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:40493
I20260810 14:40:59.596010 28031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:40493 every 8 connection(s)
I20260810 14:40:59.597002 27791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:40:59.601336 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.585692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:59775 (local address 127.15.84.62:37551)
0810 14:40:59.585862 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:59.585868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:59.586966 (+ 1098us) server_negotiation.cc:407] Connection header received
0810 14:40:59.588233 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:59.588237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:59.588302 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:59.588404 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:59.590280 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.590995 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.593009 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.593315 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.593411 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.596723 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:59.596727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:59.596730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:59.596768 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:59.598578 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:59.599312 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:59.599343 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:59.599346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:59.599486 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:59.599992 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:59.599996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:59.599998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:59.600279 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:59.600389 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:59.600974 (+ 585us) server_negotiation.cc:299] Negotiation successful
0810 14:40:59.601119 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:40:59.602250 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.586187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37551 (local address 127.15.84.3:59775)
0810 14:40:59.586738 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:59.586810 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:59.587956 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:59.588650 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:59.588660 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:59.589227 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:59.590071 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.590096 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.591201 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.591205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.592870 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.592883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.593494 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.593499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.593753 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.594559 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:59.594585 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:59.596502 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:59.598744 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:59.598752 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:59.598769 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:59.599164 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:59.599605 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:59.599609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:59.599611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:59.599845 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:59.600412 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:59.600420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:59.600716 (+ 296us) client_negotiation.cc:769] Sending connection context
0810 14:40:59.600902 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 14:40:59.601328 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":111,"threads_started":1}
I20260810 14:40:59.603677 28033 heartbeater.cc:347] Connected to a master server at 127.15.84.62:37551
I20260810 14:40:59.603850 28033 heartbeater.cc:464] Registering TS with master...
I20260810 14:40:59.604490 28033 heartbeater.cc:511] Master 127.15.84.62:37551 requested a full tablet report, sending...
I20260810 14:40:59.605199 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 27791
I20260810 14:40:59.605335 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:40:59.605648 15696 external_mini_cluster.cc:1428] Setting key e272e0f84c3f3819c7a46dcaec525ebf
I20260810 14:40:59.605677 27211 ts_manager.cc:194] Registered new tserver with Master: bd476e64d6c8463eb74768b9f6b7822f (127.15.84.3:40493)
I20260810 14:40:59.606458 27211 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:59775
I20260810 14:40:59.608503 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:40:59.613500 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.606050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:42795 (local address 127.15.84.62:37551)
0810 14:40:59.606173 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:59.606177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:59.606381 (+ 204us) server_negotiation.cc:407] Connection header received
0810 14:40:59.606497 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:59.606499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:59.606537 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:59.606624 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:59.607116 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.607822 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.609006 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.609347 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.609451 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.610556 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:59.610560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:59.610563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:59.610591 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:59.611976 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:59.612436 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:59.612438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:59.612439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:59.612506 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:59.612758 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:59.612759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:59.612760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:59.612917 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:59.612986 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:59.613276 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 14:40:59.613378 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 14:40:59.613552 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.605955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37551 (local address 127.15.84.3:42795)
0810 14:40:59.606304 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:59.606318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:59.606417 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:59.606645 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:59.606648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:59.606879 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:59.607013 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.607022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.607962 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.607965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.608887 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.608897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.609449 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.609452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.609525 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.610084 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:59.610095 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:59.610419 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:59.612085 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:59.612089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:59.612091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:59.612352 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:59.612580 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:59.612583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:59.612585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:59.612682 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:59.612995 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:59.612997 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:59.613100 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 14:40:59.613228 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 14:40:59.613376 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":103,"threads_started":1}
Aug 10 14:40:59 dist-test-slave-rmhg krb5kdc[27156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372859, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:40:59 dist-test-slave-rmhg krb5kdc[27156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:40:59 dist-test-slave-rmhg krb5kdc[27156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 14:40:59.634840 27303 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:59.628473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.15.84.62:37551)
0810 14:40:59.628604 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:59.628608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:59.628619 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:40:59.628662 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:59.628665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:59.628708 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:59.628778 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:59.629572 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.630222 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.631137 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.631363 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.631455 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.634705 (+ 3250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44784: BlockingRecv error: recv got EOF from 127.0.0.1:44784 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 14:40:59 dist-test-slave-rmhg krb5kdc[27156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372859, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
I20260810 14:40:59.644258 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.636393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44796 (local address 127.15.84.62:37551)
0810 14:40:59.636541 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:59.636546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:59.636641 (+ 95us) server_negotiation.cc:407] Connection header received
0810 14:40:59.636788 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:59.636791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:59.636843 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:59.636910 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:59.637623 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.638160 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.638963 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.639183 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.639258 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.641531 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:59.641535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:59.641537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:59.641565 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:59.642850 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:59.643222 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:59.643227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:59.643228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:59.643313 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:59.643526 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:59.643528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:59.643530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:59.643736 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:59.643839 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:59.644007 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 14:40:59.644105 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:40:59.647414 27211 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:40:59.650748 27211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:59.666361 28070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.660110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.15.84.1:43271)
0810 14:40:59.660415 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:59.660423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:59.660755 (+ 332us) server_negotiation.cc:407] Connection header received
0810 14:40:59.661001 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:59.661007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:59.661278 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:59.661416 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:59.662013 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.663262 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.664756 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.665727 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.665827 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.665982 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:59.666104 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 14:40:59.666214 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":107,"threads_started":1}
I20260810 14:40:59.666405 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.660843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.15.84.3:40493)
0810 14:40:59.661244 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:59.661257 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:59.661279 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:40:59.661360 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:59.661366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:59.661555 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:59.661700 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:59.662069 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.662889 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.664766 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.665591 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.665706 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.665912 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:59.666147 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 14:40:59.666250 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":127,"threads_started":1}
I20260810 14:40:59.666414 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.660460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:40493 (local address 127.0.0.1:59050)
0810 14:40:59.660796 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:59.660810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:59.660897 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:59.661695 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:59.661698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:59.661718 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:59.661919 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.661926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.663015 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.663019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.664597 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.664603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.665705 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.665709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.665902 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.665953 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 14:40:59.666067 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 14:40:59.666266 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":142,"thread_start_us":98,"threads_started":1}
I20260810 14:40:59.668774 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.659863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43271 (local address 127.0.0.1:33156)
0810 14:40:59.660633 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:59.660646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:59.660836 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:59.661632 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:59.661639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:59.661664 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:59.661903 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.661911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.663382 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.663385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.664589 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.664601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.665839 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.665842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.665934 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.665958 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:40:59.666073 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 14:40:59.668614 (+ 2541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":2955,"spinlock_wait_cycles":4992,"thread_start_us":113,"threads_started":1}
I20260810 14:40:59.668988 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.659859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.15.84.2:35349)
0810 14:40:59.661365 (+ 1506us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:59.661377 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:59.661401 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:40:59.661476 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:59.661480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:59.661681 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:59.661830 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:59.662071 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.663240 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.667518 (+ 4278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.668310 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.668404 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.668532 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:59.668739 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 14:40:59.668849 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1396,"thread_start_us":108,"threads_started":1}
I20260810 14:40:59.669312 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.659693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:35349 (local address 127.0.0.1:59616)
0810 14:40:59.660073 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:59.660126 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:59.660266 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:59.661871 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:59.661875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:59.661892 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:59.661979 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.661985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.663353 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.663356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.665575 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.665589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.668442 (+ 2853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.668446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.668556 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.668584 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 14:40:59.668670 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 14:40:59.668752 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":988,"thread_start_us":79,"threads_started":1}
I20260810 14:40:59.670032 27912 tablet_service.cc:1467] Processing CreateTablet for tablet b8cb2a2b77cb483e8ff108c392e4bbcb (DEFAULT_TABLE table=test-table [id=9f220d10a83445529f70ae30750b3b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:59.671368 27912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8cb2a2b77cb483e8ff108c392e4bbcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:59.671657 27444 tablet_service.cc:1467] Processing CreateTablet for tablet b8cb2a2b77cb483e8ff108c392e4bbcb (DEFAULT_TABLE table=test-table [id=9f220d10a83445529f70ae30750b3b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:59.672180 27663 tablet_service.cc:1467] Processing CreateTablet for tablet b8cb2a2b77cb483e8ff108c392e4bbcb (DEFAULT_TABLE table=test-table [id=9f220d10a83445529f70ae30750b3b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:59.672744 27444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8cb2a2b77cb483e8ff108c392e4bbcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:59.673285 27663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8cb2a2b77cb483e8ff108c392e4bbcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:59.679088 28078 tablet_bootstrap.cc:492] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331: Bootstrap starting.
I20260810 14:40:59.681640 28084 tablet_bootstrap.cc:492] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a: Bootstrap starting.
I20260810 14:40:59.681831 28078 tablet_bootstrap.cc:654] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:59.682780 28078 log.cc:824] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:59.683507 28079 tablet_bootstrap.cc:492] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f: Bootstrap starting.
I20260810 14:40:59.683992 28084 tablet_bootstrap.cc:654] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:59.684757 28084 log.cc:824] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:59.685489 28079 tablet_bootstrap.cc:654] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:59.686165 28079 log.cc:824] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:59.686841 28084 tablet_bootstrap.cc:492] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a: No bootstrap required, opened a new log
I20260810 14:40:59.687077 28084 ts_tablet_manager.cc:1397] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:40:59.687721 28078 tablet_bootstrap.cc:492] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331: No bootstrap required, opened a new log
I20260810 14:40:59.687916 28078 ts_tablet_manager.cc:1397] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260810 14:40:59.690846 28079 tablet_bootstrap.cc:492] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f: No bootstrap required, opened a new log
I20260810 14:40:59.690721 28084 raft_consensus.cc:348] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } }
I20260810 14:40:59.691042 28084 raft_consensus.cc:374] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:59.691087 28079 ts_tablet_manager.cc:1397] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260810 14:40:59.691130 28084 raft_consensus.cc:729] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d041883b7f4dfc8653f1bca560fd9a, State: Initialized, Role: FOLLOWER
I20260810 14:40:59.691695 28084 consensus_queue.cc:260] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } }
I20260810 14:40:59.691841 28078 raft_consensus.cc:348] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } }
I20260810 14:40:59.692116 28078 raft_consensus.cc:374] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:59.692194 28078 raft_consensus.cc:729] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48dd5c98ead413e881a9187a8678331, State: Initialized, Role: FOLLOWER
I20260810 14:40:59.692637 27779 heartbeater.cc:503] Master 127.15.84.62:37551 was elected leader, sending a full tablet report...
I20260810 14:40:59.692894 28084 ts_tablet_manager.cc:1428] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 14:40:59.692669 28078 consensus_queue.cc:260] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } }
I20260810 14:40:59.693560 27552 heartbeater.cc:503] Master 127.15.84.62:37551 was elected leader, sending a full tablet report...
I20260810 14:40:59.693775 28078 ts_tablet_manager.cc:1428] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 14:40:59.696195 28079 raft_consensus.cc:348] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } }
I20260810 14:40:59.696523 28079 raft_consensus.cc:374] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:59.696594 28079 raft_consensus.cc:729] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd476e64d6c8463eb74768b9f6b7822f, State: Initialized, Role: FOLLOWER
I20260810 14:40:59.697113 28079 consensus_queue.cc:260] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } }
I20260810 14:40:59.698104 28033 heartbeater.cc:503] Master 127.15.84.62:37551 was elected leader, sending a full tablet report...
I20260810 14:40:59.698472 28079 ts_tablet_manager.cc:1428] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 14:40:59.840468 28093 raft_consensus.cc:482] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:40:59.840718 28093 raft_consensus.cc:504] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } }
I20260810 14:40:59.842170 28093 leader_election.cc:288] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd476e64d6c8463eb74768b9f6b7822f (127.15.84.3:40493), f48dd5c98ead413e881a9187a8678331 (127.15.84.1:43271)
I20260810 14:40:59.847023 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.842459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43271 (local address 127.15.84.2:38809)
0810 14:40:59.842689 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:59.842706 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:59.842810 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:59.843095 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:59.843098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:59.843124 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:59.843254 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.843262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.844280 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.844284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.845854 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.845868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.846578 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.846581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.846694 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.846715 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 14:40:59.846785 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 14:40:59.846861 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260810 14:40:59.847142 28070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.842726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:38809 (local address 127.15.84.1:43271)
0810 14:40:59.842851 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:59.842856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:59.842870 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:40:59.842927 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:59.842929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:59.842981 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:59.843053 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:59.843390 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.844152 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.846002 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.846453 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.846552 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.846623 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:59.846912 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 14:40:59.847007 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:40:59.847211 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.842602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:33695 (local address 127.15.84.3:40493)
0810 14:40:59.842751 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:59.842755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:59.843232 (+ 477us) server_negotiation.cc:407] Connection header received
0810 14:40:59.843351 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:59.843354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:59.843406 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:59.843476 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:40:59.843745 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.844396 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.846055 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.846673 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.846770 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.846833 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:59.847015 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 14:40:59.847101 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 14:40:59.847558 28097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.842459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:40493 (local address 127.15.84.2:33695)
0810 14:40:59.843126 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:59.843140 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:59.843246 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:59.843496 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:59.843498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:59.843514 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:40:59.843625 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.843630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.844515 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.844519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.845917 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:59.845930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.846782 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.846785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:59.846874 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.846895 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 14:40:59.847346 (+ 451us) client_negotiation.cc:240] Negotiation successful
0810 14:40:59.847420 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":306,"thread_start_us":127,"threads_started":1}
I20260810 14:40:59.847908 27478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8cb2a2b77cb483e8ff108c392e4bbcb" candidate_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48dd5c98ead413e881a9187a8678331" is_pre_election: true
I20260810 14:40:59.848250 27478 raft_consensus.cc:2454] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d041883b7f4dfc8653f1bca560fd9a in term 0.
I20260810 14:40:59.848345 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8cb2a2b77cb483e8ff108c392e4bbcb" candidate_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd476e64d6c8463eb74768b9f6b7822f" is_pre_election: true
I20260810 14:40:59.848641 27950 raft_consensus.cc:2454] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d041883b7f4dfc8653f1bca560fd9a in term 0.
I20260810 14:40:59.848744 27592 leader_election.cc:302] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f48dd5c98ead413e881a9187a8678331, f4d041883b7f4dfc8653f1bca560fd9a; no voters:
W20260810 14:40:59.849011 28035 tablet.cc:2369] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:40:59.849045 28093 raft_consensus.cc:2781] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:40:59.849126 28093 raft_consensus.cc:482] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:40:59.849220 28093 raft_consensus.cc:3037] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:59.850333 28093 raft_consensus.cc:504] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } }
I20260810 14:40:59.850694 28093 leader_election.cc:288] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [CANDIDATE]: Term 1 election: Requested vote from peers bd476e64d6c8463eb74768b9f6b7822f (127.15.84.3:40493), f48dd5c98ead413e881a9187a8678331 (127.15.84.1:43271)
I20260810 14:40:59.851053 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8cb2a2b77cb483e8ff108c392e4bbcb" candidate_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd476e64d6c8463eb74768b9f6b7822f"
I20260810 14:40:59.851131 27478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8cb2a2b77cb483e8ff108c392e4bbcb" candidate_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48dd5c98ead413e881a9187a8678331"
I20260810 14:40:59.851176 27950 raft_consensus.cc:3037] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:59.851267 27478 raft_consensus.cc:3037] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:59.852346 27478 raft_consensus.cc:2454] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d041883b7f4dfc8653f1bca560fd9a in term 1.
I20260810 14:40:59.852641 27592 leader_election.cc:302] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f48dd5c98ead413e881a9187a8678331, f4d041883b7f4dfc8653f1bca560fd9a; no voters:
I20260810 14:40:59.852797 28093 raft_consensus.cc:2781] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:59.852851 27950 raft_consensus.cc:2454] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d041883b7f4dfc8653f1bca560fd9a in term 1.
I20260810 14:40:59.852983 28093 raft_consensus.cc:686] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [term 1 LEADER]: Becoming Leader. State: Replica: f4d041883b7f4dfc8653f1bca560fd9a, State: Running, Role: LEADER
I20260810 14:40:59.853374 28093 consensus_queue.cc:237] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } }
I20260810 14:40:59.856460 27211 catalog_manager.cc:5495] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a reported cstate change: term changed from 0 to 1, leader changed from <none> to f4d041883b7f4dfc8653f1bca560fd9a (127.15.84.2). New cstate: current_term: 1 leader_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } health_report { overall_health: UNKNOWN } } }
W20260810 14:40:59.861327 27781 tablet.cc:2369] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:40:59.882917 27553 tablet.cc:2369] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:40:59.907069 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:59.903719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.15.84.2:35349)
0810 14:40:59.903844 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:59.903847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:59.903860 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:40:59.903940 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:59.903942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:59.904002 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:59.904121 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:59.904332 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.904872 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.905720 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:59.905946 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:59.906022 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:59.906282 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:59.906770 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:59.906858 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:59.906900 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 14:40:59.906955 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:40:59.913231 27950 raft_consensus.cc:1261] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Refusing update from remote peer f4d041883b7f4dfc8653f1bca560fd9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:59.913236 27478 raft_consensus.cc:1261] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Refusing update from remote peer f4d041883b7f4dfc8653f1bca560fd9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:59.913983 28098 consensus_queue.cc:1035] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:59.914186 28093 consensus_queue.cc:1035] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:59.922804 28103 mvcc.cc:205] Tried to move back new op lower bound from 7316983234196860928 to 7316983233962344448. Current Snapshot: MvccSnapshot[applied={T|T < 7316983234196860928}]
I20260810 14:40:59.923442 28104 mvcc.cc:205] Tried to move back new op lower bound from 7316983234196860928 to 7316983233962344448. Current Snapshot: MvccSnapshot[applied={T|T < 7316983234196860928}]
I20260810 14:40:59.931457 27214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:59.931636 27214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:40:59.934186 27214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde: Sending DeleteTablet for 3 replicas of tablet b8cb2a2b77cb483e8ff108c392e4bbcb
I20260810 14:40:59.935205 27912 tablet_service.cc:1514] Processing DeleteTablet for tablet b8cb2a2b77cb483e8ff108c392e4bbcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:59 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:59050
I20260810 14:40:59.935933 28119 tablet_replica.cc:330] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f: stopping tablet replica
I20260810 14:40:59.935909 27214 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 14:40:59.935968 27444 tablet_service.cc:1514] Processing DeleteTablet for tablet b8cb2a2b77cb483e8ff108c392e4bbcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:59 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:33156
I20260810 14:40:59.936369 28119 raft_consensus.cc:2229] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 14:40:59.936430 27214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:40:59.936702 27663 tablet_service.cc:1514] Processing DeleteTablet for tablet b8cb2a2b77cb483e8ff108c392e4bbcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:40:59 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:59616
I20260810 14:40:59.936771 28119 raft_consensus.cc:2258] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:59.937250 28120 tablet_replica.cc:330] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331: stopping tablet replica
I20260810 14:40:59.937563 28120 raft_consensus.cc:2229] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:40:59.937886 28120 raft_consensus.cc:2258] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:59.938272 28122 tablet_replica.cc:330] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a: stopping tablet replica
I20260810 14:40:59.938555 28122 raft_consensus.cc:2229] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:40:59.938573 28119 ts_tablet_manager.cc:1905] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:59.938922 28122 raft_consensus.cc:2258] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:40:59.942350 28119 ts_tablet_manager.cc:1918] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:59.942472 28119 log.cc:1196] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/b8cb2a2b77cb483e8ff108c392e4bbcb
I20260810 14:40:59.942709 27444 tablet_service.cc:1467] Processing CreateTablet for tablet 788c5ed8a88047e2b6610b3389abe84e (DEFAULT_TABLE table=test-table [id=5bea08f879db4dd281f58837d2c194b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:59.942812 28119 ts_tablet_manager.cc:1939] T b8cb2a2b77cb483e8ff108c392e4bbcb P bd476e64d6c8463eb74768b9f6b7822f: Deleting consensus metadata
I20260810 14:40:59.942999 27444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788c5ed8a88047e2b6610b3389abe84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:59.945210 28078 tablet_bootstrap.cc:492] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331: Bootstrap starting.
I20260810 14:40:59.945835 27663 tablet_service.cc:1467] Processing CreateTablet for tablet 788c5ed8a88047e2b6610b3389abe84e (DEFAULT_TABLE table=test-table [id=5bea08f879db4dd281f58837d2c194b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:59.946121 27663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788c5ed8a88047e2b6610b3389abe84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:59.946211 28078 tablet_bootstrap.cc:654] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:59.947485 28078 tablet_bootstrap.cc:492] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331: No bootstrap required, opened a new log
I20260810 14:40:59.947579 28078 ts_tablet_manager.cc:1397] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:40:59.948036 28078 raft_consensus.cc:348] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:40:59.948159 28078 raft_consensus.cc:374] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:59.948205 28078 raft_consensus.cc:729] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48dd5c98ead413e881a9187a8678331, State: Initialized, Role: FOLLOWER
I20260810 14:40:59.948328 28078 consensus_queue.cc:260] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:40:59.948494 28084 tablet_bootstrap.cc:492] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a: Bootstrap starting.
I20260810 14:40:59.948730 27912 tablet_service.cc:1467] Processing CreateTablet for tablet 788c5ed8a88047e2b6610b3389abe84e (DEFAULT_TABLE table=test-table [id=5bea08f879db4dd281f58837d2c194b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:40:59.948987 27912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788c5ed8a88047e2b6610b3389abe84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:59.949221 28078 ts_tablet_manager.cc:1428] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:40:59.949537 28084 tablet_bootstrap.cc:654] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:59.950621 28122 ts_tablet_manager.cc:1905] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:59.950755 28084 tablet_bootstrap.cc:492] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a: No bootstrap required, opened a new log
I20260810 14:40:59.950850 28084 ts_tablet_manager.cc:1397] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:40:59.951170 28079 tablet_bootstrap.cc:492] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f: Bootstrap starting.
I20260810 14:40:59.951319 28084 raft_consensus.cc:348] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:40:59.951422 28084 raft_consensus.cc:374] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:59.951452 28084 raft_consensus.cc:729] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d041883b7f4dfc8653f1bca560fd9a, State: Initialized, Role: FOLLOWER
I20260810 14:40:59.951573 28084 consensus_queue.cc:260] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:40:59.951587 27195 catalog_manager.cc:4841] TS bd476e64d6c8463eb74768b9f6b7822f (127.15.84.3:40493): tablet b8cb2a2b77cb483e8ff108c392e4bbcb (table test-table [id=9f220d10a83445529f70ae30750b3b34]) successfully deleted
I20260810 14:40:59.951838 28084 ts_tablet_manager.cc:1428] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:59.952137 28079 tablet_bootstrap.cc:654] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:59.953184 28120 ts_tablet_manager.cc:1905] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:40:59.953295 28122 ts_tablet_manager.cc:1918] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:59.953388 28122 log.cc:1196] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/b8cb2a2b77cb483e8ff108c392e4bbcb
I20260810 14:40:59.953751 28122 ts_tablet_manager.cc:1939] T b8cb2a2b77cb483e8ff108c392e4bbcb P f4d041883b7f4dfc8653f1bca560fd9a: Deleting consensus metadata
I20260810 14:40:59.954962 27195 catalog_manager.cc:4841] TS f4d041883b7f4dfc8653f1bca560fd9a (127.15.84.2:35349): tablet b8cb2a2b77cb483e8ff108c392e4bbcb (table test-table [id=9f220d10a83445529f70ae30750b3b34]) successfully deleted
I20260810 14:40:59.955914 28079 tablet_bootstrap.cc:492] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f: No bootstrap required, opened a new log
I20260810 14:40:59.956014 28079 ts_tablet_manager.cc:1397] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 14:40:59.956328 28120 ts_tablet_manager.cc:1918] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:40:59.956423 28120 log.cc:1196] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/wal/wals/b8cb2a2b77cb483e8ff108c392e4bbcb
I20260810 14:40:59.956485 28079 raft_consensus.cc:348] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:40:59.956605 28079 raft_consensus.cc:374] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:59.956650 28079 raft_consensus.cc:729] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd476e64d6c8463eb74768b9f6b7822f, State: Initialized, Role: FOLLOWER
I20260810 14:40:59.956699 28120 ts_tablet_manager.cc:1939] T b8cb2a2b77cb483e8ff108c392e4bbcb P f48dd5c98ead413e881a9187a8678331: Deleting consensus metadata
I20260810 14:40:59.956781 28079 consensus_queue.cc:260] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:40:59.957074 28079 ts_tablet_manager.cc:1428] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:40:59.958149 27194 catalog_manager.cc:4841] TS f48dd5c98ead413e881a9187a8678331 (127.15.84.1:43271): tablet b8cb2a2b77cb483e8ff108c392e4bbcb (table test-table [id=9f220d10a83445529f70ae30750b3b34]) successfully deleted
I20260810 14:41:00.178771 28094 raft_consensus.cc:482] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:41:00.178956 28094 raft_consensus.cc:504] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.180145 28094 leader_election.cc:288] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4d041883b7f4dfc8653f1bca560fd9a (127.15.84.2:35349), bd476e64d6c8463eb74768b9f6b7822f (127.15.84.3:40493)
I20260810 14:41:00.185488 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.180597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:44235 (local address 127.15.84.2:35349)
0810 14:41:00.180777 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:00.180782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:00.180939 (+ 157us) server_negotiation.cc:407] Connection header received
0810 14:41:00.181113 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:00.181117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:00.181212 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:00.181287 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:41:00.181562 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.182322 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.184085 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.184766 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.184874 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.184947 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:00.185265 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 14:41:00.185347 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 14:41:00.185488 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.181082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:46237 (local address 127.15.84.3:40493)
0810 14:41:00.181233 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:00.181238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:00.181421 (+ 183us) server_negotiation.cc:407] Connection header received
0810 14:41:00.181515 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:00.181518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:00.181567 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:00.181643 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:41:00.181883 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.182554 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.184086 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.184766 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.184874 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.184946 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:00.185265 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 14:41:00.185347 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:41:00.185547 28134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.180937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:40493 (local address 127.15.84.1:46237)
0810 14:41:00.181316 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:00.181330 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:00.181424 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:00.181666 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:00.181669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:00.181684 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:41:00.181782 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.181788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.182672 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.182675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.183952 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.183965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.184903 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.184907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.185111 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.185172 (+ 61us) client_negotiation.cc:769] Sending connection context
0810 14:41:00.185238 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 14:41:00.185356 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":97,"thread_start_us":102,"threads_started":1}
I20260810 14:41:00.185556 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.180506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:35349 (local address 127.15.84.1:44235)
0810 14:41:00.180820 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:00.180833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:00.180971 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:00.181309 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:00.181313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:00.181334 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:41:00.181458 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.181465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.182444 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.182447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.183953 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.183965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.184903 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.184907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.185084 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.185139 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 14:41:00.185246 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 14:41:00.185356 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":132,"thread_start_us":90,"threads_started":1}
I20260810 14:41:00.186394 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788c5ed8a88047e2b6610b3389abe84e" candidate_uuid: "f48dd5c98ead413e881a9187a8678331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd476e64d6c8463eb74768b9f6b7822f" is_pre_election: true
I20260810 14:41:00.186434 27707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788c5ed8a88047e2b6610b3389abe84e" candidate_uuid: "f48dd5c98ead413e881a9187a8678331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" is_pre_election: true
I20260810 14:41:00.186573 27950 raft_consensus.cc:2454] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48dd5c98ead413e881a9187a8678331 in term 0.
I20260810 14:41:00.186663 27707 raft_consensus.cc:2454] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48dd5c98ead413e881a9187a8678331 in term 0.
I20260810 14:41:00.186987 27362 leader_election.cc:302] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd476e64d6c8463eb74768b9f6b7822f, f48dd5c98ead413e881a9187a8678331; no voters:
I20260810 14:41:00.187253 28094 raft_consensus.cc:2781] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:41:00.187310 28094 raft_consensus.cc:482] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:41:00.187335 28094 raft_consensus.cc:3037] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:00.188398 28094 raft_consensus.cc:504] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.188758 28094 leader_election.cc:288] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [CANDIDATE]: Term 1 election: Requested vote from peers f4d041883b7f4dfc8653f1bca560fd9a (127.15.84.2:35349), bd476e64d6c8463eb74768b9f6b7822f (127.15.84.3:40493)
I20260810 14:41:00.189097 27707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788c5ed8a88047e2b6610b3389abe84e" candidate_uuid: "f48dd5c98ead413e881a9187a8678331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d041883b7f4dfc8653f1bca560fd9a"
I20260810 14:41:00.189214 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788c5ed8a88047e2b6610b3389abe84e" candidate_uuid: "f48dd5c98ead413e881a9187a8678331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd476e64d6c8463eb74768b9f6b7822f"
I20260810 14:41:00.189284 27707 raft_consensus.cc:3037] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:00.189332 27950 raft_consensus.cc:3037] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:00.190477 27707 raft_consensus.cc:2454] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48dd5c98ead413e881a9187a8678331 in term 1.
I20260810 14:41:00.190477 27950 raft_consensus.cc:2454] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48dd5c98ead413e881a9187a8678331 in term 1.
I20260810 14:41:00.190819 27362 leader_election.cc:302] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f48dd5c98ead413e881a9187a8678331, f4d041883b7f4dfc8653f1bca560fd9a; no voters:
I20260810 14:41:00.191036 28094 raft_consensus.cc:2781] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:41:00.191257 28094 raft_consensus.cc:686] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [term 1 LEADER]: Becoming Leader. State: Replica: f48dd5c98ead413e881a9187a8678331, State: Running, Role: LEADER
I20260810 14:41:00.191540 28094 consensus_queue.cc:237] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.194011 27214 catalog_manager.cc:5495] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 reported cstate change: term changed from 0 to 1, leader changed from <none> to f48dd5c98ead413e881a9187a8678331 (127.15.84.1). New cstate: current_term: 1 leader_uuid: "f48dd5c98ead413e881a9187a8678331" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } health_report { overall_health: UNKNOWN } } }
Aug 10 14:41:00 dist-test-slave-rmhg krb5kdc[27156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372858, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.62@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
I20260810 14:41:00.268574 28146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.259244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37551 (local address 127.0.0.1:59198)
0810 14:41:00.259596 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:00.259609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:00.259713 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:00.260015 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:00.260017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:00.260442 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:00.260545 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.260550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.261444 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.261447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.262052 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.262061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.262502 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.262504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.262566 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.263148 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:00.263161 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:00.264855 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:41:00.266615 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:41:00.266629 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:41:00.266636 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:41:00.267004 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:41:00.267380 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:41:00.267384 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:41:00.267386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:41:00.267482 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:41:00.267914 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:41:00.267923 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:41:00.268052 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 14:41:00.268155 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 14:41:00.268348 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":133,"threads_started":1}
I20260810 14:41:00.268574 28147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.259328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.15.84.62:37551)
0810 14:41:00.259635 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:00.259643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:00.259663 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:41:00.259837 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:00.259840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:00.259889 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:00.259988 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:00.260679 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.261336 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.262157 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.262404 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.262494 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.265012 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:00.265016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:00.265018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:41:00.265042 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:00.266467 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:00.267137 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:00.267141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:00.267143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:00.267238 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:00.267578 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:00.267582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:00.267584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:00.267800 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:00.267894 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:00.268219 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 14:41:00.268429 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":232,"thread_start_us":128,"threads_started":1}
I20260810 14:41:00.273351 27214 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:59198:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 14:41:00.274183 27214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:41:00.279397 27912 tablet_service.cc:1467] Processing CreateTablet for tablet 652ac998491b4ee5ae8276c5cf5c1d42 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70169e0026b140e2ad40719c5cb84516]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:41:00.279412 27444 tablet_service.cc:1467] Processing CreateTablet for tablet 652ac998491b4ee5ae8276c5cf5c1d42 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70169e0026b140e2ad40719c5cb84516]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:41:00.279402 27663 tablet_service.cc:1467] Processing CreateTablet for tablet 652ac998491b4ee5ae8276c5cf5c1d42 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70169e0026b140e2ad40719c5cb84516]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:41:00.279873 27663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652ac998491b4ee5ae8276c5cf5c1d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:00.279873 27444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652ac998491b4ee5ae8276c5cf5c1d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:00.279882 27912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652ac998491b4ee5ae8276c5cf5c1d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:00.282784 28079 tablet_bootstrap.cc:492] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f: Bootstrap starting.
I20260810 14:41:00.282850 28078 tablet_bootstrap.cc:492] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331: Bootstrap starting.
I20260810 14:41:00.282848 28084 tablet_bootstrap.cc:492] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a: Bootstrap starting.
I20260810 14:41:00.283679 28079 tablet_bootstrap.cc:654] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:00.283758 28084 tablet_bootstrap.cc:654] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:00.283777 28078 tablet_bootstrap.cc:654] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:00.285087 28079 tablet_bootstrap.cc:492] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f: No bootstrap required, opened a new log
I20260810 14:41:00.285220 28079 ts_tablet_manager.cc:1397] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:41:00.285351 28084 tablet_bootstrap.cc:492] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a: No bootstrap required, opened a new log
I20260810 14:41:00.285440 28084 ts_tablet_manager.cc:1397] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:41:00.285688 28079 raft_consensus.cc:348] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.285784 28079 raft_consensus.cc:374] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:00.285804 28079 raft_consensus.cc:729] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd476e64d6c8463eb74768b9f6b7822f, State: Initialized, Role: FOLLOWER
I20260810 14:41:00.285825 28078 tablet_bootstrap.cc:492] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331: No bootstrap required, opened a new log
I20260810 14:41:00.285912 28078 ts_tablet_manager.cc:1397] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:41:00.285863 28084 raft_consensus.cc:348] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.285912 28079 consensus_queue.cc:260] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.285969 28084 raft_consensus.cc:374] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:00.286001 28084 raft_consensus.cc:729] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d041883b7f4dfc8653f1bca560fd9a, State: Initialized, Role: FOLLOWER
I20260810 14:41:00.286123 28084 consensus_queue.cc:260] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.286151 28096 tablet_replica.cc:439] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } }
I20260810 14:41:00.286311 28096 tablet_replica.cc:442] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:41:00.286392 28084 ts_tablet_manager.cc:1428] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:41:00.286345 28078 raft_consensus.cc:348] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.286458 28078 raft_consensus.cc:374] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:00.286489 28078 raft_consensus.cc:729] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48dd5c98ead413e881a9187a8678331, State: Initialized, Role: FOLLOWER
I20260810 14:41:00.286604 28079 ts_tablet_manager.cc:1428] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:41:00.286604 28078 consensus_queue.cc:260] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.286665 28110 tablet_replica.cc:439] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } }
I20260810 14:41:00.286795 28110 tablet_replica.cc:442] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:41:00.286861 28078 ts_tablet_manager.cc:1428] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:41:00.287173 28094 tablet_replica.cc:439] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } }
I20260810 14:41:00.287298 28094 tablet_replica.cc:442] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:41:00.443567 28096 raft_consensus.cc:482] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:41:00.443732 28096 raft_consensus.cc:504] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.445008 28096 leader_election.cc:288] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f48dd5c98ead413e881a9187a8678331 (127.15.84.1:43271), f4d041883b7f4dfc8653f1bca560fd9a (127.15.84.2:35349)
I20260810 14:41:00.448887 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.445323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:35349 (local address 127.15.84.3:36487)
0810 14:41:00.445660 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:00.445675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:00.445799 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:00.446035 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:00.446037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:00.446053 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:41:00.446155 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.446161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.446868 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.446871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.447804 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.447814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.448450 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.448453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.448554 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.448574 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 14:41:00.448640 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 14:41:00.448720 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260810 14:41:00.448872 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.445504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:36487 (local address 127.15.84.2:35349)
0810 14:41:00.445664 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:00.445692 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:00.445739 (+ 47us) server_negotiation.cc:407] Connection header received
0810 14:41:00.445900 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:00.445903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:00.445952 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:00.446017 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:41:00.446260 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.446766 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.447925 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.448335 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.448427 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.448496 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:00.448664 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 14:41:00.448751 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:41:00.448971 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.445414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:58641 (local address 127.15.84.1:43271)
0810 14:41:00.445699 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:00.445703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:00.445845 (+ 142us) server_negotiation.cc:407] Connection header received
0810 14:41:00.445997 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:00.446000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:00.446044 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:00.446111 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:41:00.446358 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.446888 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.447948 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.448419 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.448514 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.448581 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:00.448763 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 14:41:00.448849 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":99,"threads_started":1}
I20260810 14:41:00.449352 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.445323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43271 (local address 127.15.84.3:58641)
0810 14:41:00.445755 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:00.445767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:00.445881 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:00.446128 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:00.446131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:00.446147 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:41:00.446259 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.446265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.446995 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.446998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.447829 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.447839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.448539 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.448543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.448652 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.448672 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 14:41:00.448736 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 14:41:00.448809 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":60,"thread_start_us":63,"threads_started":1}
I20260810 14:41:00.449431 27707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652ac998491b4ee5ae8276c5cf5c1d42" candidate_uuid: "bd476e64d6c8463eb74768b9f6b7822f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" is_pre_election: true
I20260810 14:41:00.449611 27707 raft_consensus.cc:2454] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd476e64d6c8463eb74768b9f6b7822f in term 0.
I20260810 14:41:00.449859 27478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652ac998491b4ee5ae8276c5cf5c1d42" candidate_uuid: "bd476e64d6c8463eb74768b9f6b7822f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48dd5c98ead413e881a9187a8678331" is_pre_election: true
I20260810 14:41:00.450042 27478 raft_consensus.cc:2454] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd476e64d6c8463eb74768b9f6b7822f in term 0.
I20260810 14:41:00.450011 27820 leader_election.cc:302] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd476e64d6c8463eb74768b9f6b7822f, f4d041883b7f4dfc8653f1bca560fd9a; no voters:
I20260810 14:41:00.450377 28096 raft_consensus.cc:2781] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:41:00.450436 28096 raft_consensus.cc:482] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:41:00.450475 28096 raft_consensus.cc:3037] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:00.451521 28096 raft_consensus.cc:504] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.451870 28096 leader_election.cc:288] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [CANDIDATE]: Term 1 election: Requested vote from peers f48dd5c98ead413e881a9187a8678331 (127.15.84.1:43271), f4d041883b7f4dfc8653f1bca560fd9a (127.15.84.2:35349)
I20260810 14:41:00.452270 27707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652ac998491b4ee5ae8276c5cf5c1d42" candidate_uuid: "bd476e64d6c8463eb74768b9f6b7822f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d041883b7f4dfc8653f1bca560fd9a"
I20260810 14:41:00.452281 27478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652ac998491b4ee5ae8276c5cf5c1d42" candidate_uuid: "bd476e64d6c8463eb74768b9f6b7822f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48dd5c98ead413e881a9187a8678331"
I20260810 14:41:00.452382 27478 raft_consensus.cc:3037] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:00.452390 27707 raft_consensus.cc:3037] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:00.453393 27707 raft_consensus.cc:2454] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd476e64d6c8463eb74768b9f6b7822f in term 1.
I20260810 14:41:00.453397 27478 raft_consensus.cc:2454] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd476e64d6c8463eb74768b9f6b7822f in term 1.
I20260810 14:41:00.453756 27817 leader_election.cc:302] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: bd476e64d6c8463eb74768b9f6b7822f, f48dd5c98ead413e881a9187a8678331, f4d041883b7f4dfc8653f1bca560fd9a; no voters:
I20260810 14:41:00.453930 28096 raft_consensus.cc:2781] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:41:00.454090 28096 raft_consensus.cc:686] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [term 1 LEADER]: Becoming Leader. State: Replica: bd476e64d6c8463eb74768b9f6b7822f, State: Running, Role: LEADER
I20260810 14:41:00.454355 28096 consensus_queue.cc:237] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.455758 28163 tablet_replica.cc:439] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f: TxnStatusTablet state changed. Reason: New leader bd476e64d6c8463eb74768b9f6b7822f. Latest consensus state: current_term: 1 leader_uuid: "bd476e64d6c8463eb74768b9f6b7822f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } }
I20260810 14:41:00.455878 28163 tablet_replica.cc:442] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:41:00.456100 28165 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 14:41:00.456265 28165 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 14:41:00.457029 27214 catalog_manager.cc:5495] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f reported cstate change: term changed from 0 to 1, leader changed from <none> to bd476e64d6c8463eb74768b9f6b7822f (127.15.84.3). New cstate: current_term: 1 leader_uuid: "bd476e64d6c8463eb74768b9f6b7822f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } health_report { overall_health: HEALTHY } } }
I20260810 14:41:00.478979 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.475337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59060 (local address 127.15.84.3:40493)
0810 14:41:00.475451 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:00.475455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:00.475469 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:41:00.475686 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:00.475689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:00.475747 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:00.475833 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:41:00.476071 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.476588 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.477488 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.477708 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.477798 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.478095 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:41:00.478556 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:41:00.478668 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:00.478764 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 14:41:00.478825 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:41:00.478979 28146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.475256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:40493 (local address 127.0.0.1:59060)
0810 14:41:00.475409 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:00.475425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:00.475536 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:00.475856 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:00.475859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:00.475868 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:41:00.475973 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.475978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.476698 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.476702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.477373 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.477383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.477808 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.477812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.477902 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.477952 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:41:00.478652 (+ 700us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:41:00.478679 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 14:41:00.478745 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 14:41:00.478846 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 14:41:00.482213 27478 raft_consensus.cc:1261] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Refusing update from remote peer bd476e64d6c8463eb74768b9f6b7822f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:41:00.482383 27707 raft_consensus.cc:1261] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Refusing update from remote peer bd476e64d6c8463eb74768b9f6b7822f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:41:00.482833 28096 consensus_queue.cc:1035] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:41:00.483043 28163 consensus_queue.cc:1035] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:41:00.484656 28094 tablet_replica.cc:439] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331: TxnStatusTablet state changed. Reason: New leader bd476e64d6c8463eb74768b9f6b7822f. Latest consensus state: current_term: 1 leader_uuid: "bd476e64d6c8463eb74768b9f6b7822f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } }
I20260810 14:41:00.484810 28094 tablet_replica.cc:442] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:41:00.484966 28110 tablet_replica.cc:439] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a: TxnStatusTablet state changed. Reason: New leader bd476e64d6c8463eb74768b9f6b7822f. Latest consensus state: current_term: 1 leader_uuid: "bd476e64d6c8463eb74768b9f6b7822f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } }
I20260810 14:41:00.485081 28110 tablet_replica.cc:442] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:41:00.487955 28163 tablet_replica.cc:439] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd476e64d6c8463eb74768b9f6b7822f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } }
I20260810 14:41:00.488083 28163 tablet_replica.cc:442] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:41:00.488096 28096 tablet_replica.cc:439] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd476e64d6c8463eb74768b9f6b7822f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } }
I20260810 14:41:00.488217 28096 tablet_replica.cc:442] T 652ac998491b4ee5ae8276c5cf5c1d42 P bd476e64d6c8463eb74768b9f6b7822f: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:41:00.488324 28094 tablet_replica.cc:439] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd476e64d6c8463eb74768b9f6b7822f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } }
I20260810 14:41:00.488425 28094 tablet_replica.cc:442] T 652ac998491b4ee5ae8276c5cf5c1d42 P f48dd5c98ead413e881a9187a8678331: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:41:00.489670 28110 tablet_replica.cc:439] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd476e64d6c8463eb74768b9f6b7822f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } } }
I20260810 14:41:00.489795 28110 tablet_replica.cc:442] T 652ac998491b4ee5ae8276c5cf5c1d42 P f4d041883b7f4dfc8653f1bca560fd9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:41:00.490823 28172 mvcc.cc:205] Tried to move back new op lower bound from 7316983236529348608 to 7316983236423852032. Current Snapshot: MvccSnapshot[applied={T|T < 7316983236529348608 or (T in {7316983236529348608})}]
I20260810 14:41:00.498798 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.494736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33168 (local address 127.15.84.1:43271)
0810 14:41:00.494865 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:00.494870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:00.495032 (+ 162us) server_negotiation.cc:407] Connection header received
0810 14:41:00.495177 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:00.495181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:00.495253 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:00.495363 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:41:00.495564 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.496364 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.497255 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.497483 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.497572 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.497875 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:41:00.498396 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:41:00.498508 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:00.498584 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 14:41:00.498649 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 14:41:00.509907 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.506449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:40493 (local address 127.15.84.1:53967)
0810 14:41:00.506782 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:00.506794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:00.506898 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:00.507134 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:00.507137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:00.507145 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:41:00.507278 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.507284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.507994 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.507997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.508632 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.508641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.509117 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.509121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.509251 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.509302 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:41:00.509609 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:41:00.509616 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:41:00.509682 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 14:41:00.509763 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":110,"threads_started":1}
I20260810 14:41:00.509891 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.506547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:53967 (local address 127.15.84.3:40493)
0810 14:41:00.506674 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:00.506679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:00.506862 (+ 183us) server_negotiation.cc:407] Connection header received
0810 14:41:00.507003 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:00.507006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:00.507056 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:00.507117 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:41:00.507382 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.507893 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.508759 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.509012 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.509102 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.509441 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:41:00.509537 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:41:00.509595 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:00.509691 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 14:41:00.509744 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:41:00.517225 27950 raft_consensus.cc:1261] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Refusing update from remote peer f48dd5c98ead413e881a9187a8678331: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:41:00.517334 27707 raft_consensus.cc:1261] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Refusing update from remote peer f48dd5c98ead413e881a9187a8678331: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:41:00.517815 28135 consensus_queue.cc:1035] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:41:00.518208 28094 consensus_queue.cc:1035] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:41:00.544978 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.540484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:53035 (local address 127.15.84.1:43271)
0810 14:41:00.540614 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:00.540619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:00.540883 (+ 264us) server_negotiation.cc:407] Connection header received
0810 14:41:00.541016 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:00.541019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:00.541081 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:00.541190 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:41:00.541713 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.542505 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.543812 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.544047 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.544133 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.544489 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:41:00.544594 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:41:00.544667 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:00.544764 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 14:41:00.544834 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:41:00.545017 28203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.540409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43271 (local address 127.15.84.3:53035)
0810 14:41:00.540788 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:00.540804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:00.540927 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:00.541205 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:00.541208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:00.541228 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:41:00.541552 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.541561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.542668 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.542671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.543691 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:00.543702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.544153 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.544157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:00.544264 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.544329 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:41:00.544675 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:41:00.544681 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:41:00.544737 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 14:41:00.544832 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":137,"threads_started":1}
I20260810 14:41:00.568195 27214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:41:00.568358 27214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:41:00.570572 27214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde: Sending DeleteTablet for 3 replicas of tablet 788c5ed8a88047e2b6610b3389abe84e
I20260810 14:41:00.571281 27444 tablet_service.cc:1514] Processing DeleteTablet for tablet 788c5ed8a88047e2b6610b3389abe84e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:41:00 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:33156
I20260810 14:41:00.571403 27663 tablet_service.cc:1514] Processing DeleteTablet for tablet 788c5ed8a88047e2b6610b3389abe84e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:41:00 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:59616
I20260810 14:41:00.571892 27912 tablet_service.cc:1514] Processing DeleteTablet for tablet 788c5ed8a88047e2b6610b3389abe84e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:41:00 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:59050
I20260810 14:41:00.572467 28206 tablet_replica.cc:330] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331: stopping tablet replica
I20260810 14:41:00.572669 28206 raft_consensus.cc:2229] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:41:00.573336 28206 raft_consensus.cc:2258] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:41:00.573693 28208 tablet_replica.cc:330] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a: stopping tablet replica
I20260810 14:41:00.573941 28209 tablet_replica.cc:330] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f: stopping tablet replica
I20260810 14:41:00.574113 28209 raft_consensus.cc:2229] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:41:00.574220 28206 ts_tablet_manager.cc:1905] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:41:00.574349 28208 raft_consensus.cc:2229] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:41:00.574534 28209 raft_consensus.cc:2258] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:41:00.574712 28208 raft_consensus.cc:2258] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:41:00.575635 28209 ts_tablet_manager.cc:1905] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:41:00.577795 28206 ts_tablet_manager.cc:1918] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:41:00.577893 28206 log.cc:1196] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/wal/wals/788c5ed8a88047e2b6610b3389abe84e
I20260810 14:41:00.578157 28206 ts_tablet_manager.cc:1939] T 788c5ed8a88047e2b6610b3389abe84e P f48dd5c98ead413e881a9187a8678331: Deleting consensus metadata
I20260810 14:41:00.578802 28209 ts_tablet_manager.cc:1918] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:41:00.578969 27194 catalog_manager.cc:4841] TS f48dd5c98ead413e881a9187a8678331 (127.15.84.1:43271): tablet 788c5ed8a88047e2b6610b3389abe84e (table test-table [id=5bea08f879db4dd281f58837d2c194b6]) successfully deleted
I20260810 14:41:00.579033 28209 log.cc:1196] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/788c5ed8a88047e2b6610b3389abe84e
I20260810 14:41:00.579344 28209 ts_tablet_manager.cc:1939] T 788c5ed8a88047e2b6610b3389abe84e P bd476e64d6c8463eb74768b9f6b7822f: Deleting consensus metadata
I20260810 14:41:00.579959 27195 catalog_manager.cc:4841] TS bd476e64d6c8463eb74768b9f6b7822f (127.15.84.3:40493): tablet 788c5ed8a88047e2b6610b3389abe84e (table test-table [id=5bea08f879db4dd281f58837d2c194b6]) successfully deleted
I20260810 14:41:00.584339 28208 ts_tablet_manager.cc:1905] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:41:00.587683 28208 ts_tablet_manager.cc:1918] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:41:00.587779 28208 log.cc:1196] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/788c5ed8a88047e2b6610b3389abe84e
I20260810 14:41:00.587993 28208 ts_tablet_manager.cc:1939] T 788c5ed8a88047e2b6610b3389abe84e P f4d041883b7f4dfc8653f1bca560fd9a: Deleting consensus metadata
I20260810 14:41:00.588815 27195 catalog_manager.cc:4841] TS f4d041883b7f4dfc8653f1bca560fd9a (127.15.84.2:35349): tablet 788c5ed8a88047e2b6610b3389abe84e (table test-table [id=5bea08f879db4dd281f58837d2c194b6]) successfully deleted
Aug 10 14:41:00 dist-test-slave-rmhg krb5kdc[27156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:41:00 dist-test-slave-rmhg krb5kdc[27156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:00 dist-test-slave-rmhg krb5kdc[27156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 14:41:00.610872 28147 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:00.602498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.15.84.62:37551)
0810 14:41:00.602632 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:00.602636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:00.603290 (+ 654us) server_negotiation.cc:407] Connection header received
0810 14:41:00.603479 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:00.603483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:00.603531 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:00.603598 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:00.604653 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.605534 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.606733 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.607095 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.607165 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.610733 (+ 3568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59206: BlockingRecv error: recv got EOF from 127.0.0.1:59206 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 14:41:00 dist-test-slave-rmhg krb5kdc[27156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM
I20260810 14:41:00.624255 28147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:00.613694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.15.84.62:37551)
0810 14:41:00.613816 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:00.613820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:00.614385 (+ 565us) server_negotiation.cc:407] Connection header received
0810 14:41:00.614569 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:00.614573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:00.614619 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:00.614666 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:00.615505 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.616373 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.617733 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:00.618076 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:00.618143 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:00.621103 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:00.621108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:00.621111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:41:00.621143 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:00.622657 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:00.623067 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:00.623070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:00.623071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:00.623151 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:00.623411 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:00.623415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:00.623417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:00.623607 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:00.623726 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:00.623964 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 14:41:00.624087 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:41:00.626947 27214 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:41:00.627414 27214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:41:00.632588 27663 tablet_service.cc:1467] Processing CreateTablet for tablet dfa90bcfddc34c75972cc9398ee18ceb (DEFAULT_TABLE table=test-table [id=80f290c727bf4efc8fcb35ddcbd0ecbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:41:00.632568 27444 tablet_service.cc:1467] Processing CreateTablet for tablet dfa90bcfddc34c75972cc9398ee18ceb (DEFAULT_TABLE table=test-table [id=80f290c727bf4efc8fcb35ddcbd0ecbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:41:00.632916 27444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa90bcfddc34c75972cc9398ee18ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:00.632916 27663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa90bcfddc34c75972cc9398ee18ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:00.635462 28084 tablet_bootstrap.cc:492] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a: Bootstrap starting.
I20260810 14:41:00.635768 27912 tablet_service.cc:1467] Processing CreateTablet for tablet dfa90bcfddc34c75972cc9398ee18ceb (DEFAULT_TABLE table=test-table [id=80f290c727bf4efc8fcb35ddcbd0ecbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:41:00.636073 27912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa90bcfddc34c75972cc9398ee18ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:00.636425 28084 tablet_bootstrap.cc:654] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:00.638146 28079 tablet_bootstrap.cc:492] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f: Bootstrap starting.
I20260810 14:41:00.639047 28079 tablet_bootstrap.cc:654] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:00.639632 28078 tablet_bootstrap.cc:492] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331: Bootstrap starting.
I20260810 14:41:00.640471 28078 tablet_bootstrap.cc:654] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:00.641893 28078 tablet_bootstrap.cc:492] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331: No bootstrap required, opened a new log
I20260810 14:41:00.641984 28078 ts_tablet_manager.cc:1397] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:41:00.642439 28078 raft_consensus.cc:348] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.642580 28078 raft_consensus.cc:374] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:00.642617 28078 raft_consensus.cc:729] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48dd5c98ead413e881a9187a8678331, State: Initialized, Role: FOLLOWER
I20260810 14:41:00.642750 28078 consensus_queue.cc:260] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.643020 28078 ts_tablet_manager.cc:1428] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:41:00.643774 28079 tablet_bootstrap.cc:492] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f: No bootstrap required, opened a new log
I20260810 14:41:00.643870 28079 ts_tablet_manager.cc:1397] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260810 14:41:00.644284 28079 raft_consensus.cc:348] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.644397 28079 raft_consensus.cc:374] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:00.644428 28079 raft_consensus.cc:729] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd476e64d6c8463eb74768b9f6b7822f, State: Initialized, Role: FOLLOWER
I20260810 14:41:00.644568 28079 consensus_queue.cc:260] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.644791 28084 tablet_bootstrap.cc:492] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a: No bootstrap required, opened a new log
I20260810 14:41:00.644817 28079 ts_tablet_manager.cc:1428] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 14:41:00.644908 28084 ts_tablet_manager.cc:1397] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260810 14:41:00.645391 28084 raft_consensus.cc:348] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.645516 28084 raft_consensus.cc:374] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:00.645552 28084 raft_consensus.cc:729] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d041883b7f4dfc8653f1bca560fd9a, State: Initialized, Role: FOLLOWER
I20260810 14:41:00.645673 28084 consensus_queue.cc:260] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:00.645962 28084 ts_tablet_manager.cc:1428] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:41:01.002620 28094 raft_consensus.cc:482] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:41:01.002795 28094 raft_consensus.cc:504] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:01.003240 28094 leader_election.cc:288] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4d041883b7f4dfc8653f1bca560fd9a (127.15.84.2:35349), bd476e64d6c8463eb74768b9f6b7822f (127.15.84.3:40493)
I20260810 14:41:01.003847 27707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa90bcfddc34c75972cc9398ee18ceb" candidate_uuid: "f48dd5c98ead413e881a9187a8678331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" is_pre_election: true
I20260810 14:41:01.003846 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa90bcfddc34c75972cc9398ee18ceb" candidate_uuid: "f48dd5c98ead413e881a9187a8678331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd476e64d6c8463eb74768b9f6b7822f" is_pre_election: true
I20260810 14:41:01.004000 27950 raft_consensus.cc:2454] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48dd5c98ead413e881a9187a8678331 in term 0.
I20260810 14:41:01.004009 27707 raft_consensus.cc:2454] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48dd5c98ead413e881a9187a8678331 in term 0.
I20260810 14:41:01.004367 27362 leader_election.cc:302] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd476e64d6c8463eb74768b9f6b7822f, f48dd5c98ead413e881a9187a8678331; no voters:
I20260810 14:41:01.004604 28094 raft_consensus.cc:2781] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:41:01.004709 28094 raft_consensus.cc:482] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:41:01.004753 28094 raft_consensus.cc:3037] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:01.005833 28094 raft_consensus.cc:504] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:01.006207 28094 leader_election.cc:288] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [CANDIDATE]: Term 1 election: Requested vote from peers f4d041883b7f4dfc8653f1bca560fd9a (127.15.84.2:35349), bd476e64d6c8463eb74768b9f6b7822f (127.15.84.3:40493)
I20260810 14:41:01.006628 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa90bcfddc34c75972cc9398ee18ceb" candidate_uuid: "f48dd5c98ead413e881a9187a8678331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd476e64d6c8463eb74768b9f6b7822f"
I20260810 14:41:01.006633 27707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa90bcfddc34c75972cc9398ee18ceb" candidate_uuid: "f48dd5c98ead413e881a9187a8678331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d041883b7f4dfc8653f1bca560fd9a"
I20260810 14:41:01.006772 27707 raft_consensus.cc:3037] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:01.006772 27950 raft_consensus.cc:3037] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:01.007965 27707 raft_consensus.cc:2454] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48dd5c98ead413e881a9187a8678331 in term 1.
I20260810 14:41:01.007966 27950 raft_consensus.cc:2454] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48dd5c98ead413e881a9187a8678331 in term 1.
I20260810 14:41:01.008280 27362 leader_election.cc:302] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd476e64d6c8463eb74768b9f6b7822f, f48dd5c98ead413e881a9187a8678331; no voters:
I20260810 14:41:01.008455 28094 raft_consensus.cc:2781] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:41:01.008548 28094 raft_consensus.cc:686] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [term 1 LEADER]: Becoming Leader. State: Replica: f48dd5c98ead413e881a9187a8678331, State: Running, Role: LEADER
I20260810 14:41:01.008688 28094 consensus_queue.cc:237] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:01.010365 27214 catalog_manager.cc:5495] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 reported cstate change: term changed from 0 to 1, leader changed from <none> to f48dd5c98ead413e881a9187a8678331 (127.15.84.1). New cstate: current_term: 1 leader_uuid: "f48dd5c98ead413e881a9187a8678331" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } health_report { overall_health: UNKNOWN } } }
I20260810 14:41:01.101677 28232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:01.098549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.15.84.1:43271)
0810 14:41:01.098810 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:01.098814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:01.098828 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:41:01.098878 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:01.098880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:01.098926 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:01.099008 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:41:01.099227 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:01.099755 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:01.100593 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:01.100827 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:01.100909 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:01.101184 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:41:01.101272 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:41:01.101338 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:01.101464 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 14:41:01.101521 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260810 14:41:01.103840 27950 raft_consensus.cc:1261] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Refusing update from remote peer f48dd5c98ead413e881a9187a8678331: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:41:01.103910 27707 raft_consensus.cc:1261] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Refusing update from remote peer f48dd5c98ead413e881a9187a8678331: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:41:01.104311 28094 consensus_queue.cc:1035] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:41:01.104524 28135 consensus_queue.cc:1035] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:41:01.112323 27210 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:41:01.112466 27210 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:41:01.114432 27210 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde: Sending DeleteTablet for 3 replicas of tablet dfa90bcfddc34c75972cc9398ee18ceb
I20260810 14:41:01.115064 27444 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa90bcfddc34c75972cc9398ee18ceb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:41:01 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:33156
I20260810 14:41:01.115131 27663 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa90bcfddc34c75972cc9398ee18ceb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:41:01 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:59616
I20260810 14:41:01.115190 27912 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa90bcfddc34c75972cc9398ee18ceb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:41:01 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:59050
I20260810 14:41:01.115463 28253 tablet_replica.cc:330] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a: stopping tablet replica
I20260810 14:41:01.115599 28253 raft_consensus.cc:2229] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:41:01.115743 28253 raft_consensus.cc:2258] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:41:01.115878 28250 tablet_replica.cc:330] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f: stopping tablet replica
I20260810 14:41:01.116008 28250 raft_consensus.cc:2229] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:41:01.116154 28250 raft_consensus.cc:2258] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:41:01.116178 27210 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 14:41:01.116415 28249 tablet_replica.cc:330] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331: stopping tablet replica
I20260810 14:41:01.116547 28249 raft_consensus.cc:2229] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [term 1 LEADER]: Raft consensus shutting down.
W20260810 14:41:01.116621 27210 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:41:01.116863 28249 raft_consensus.cc:2258] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:41:01.117185 28250 ts_tablet_manager.cc:1905] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:41:01.117609 28249 ts_tablet_manager.cc:1905] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:41:01.118484 28253 ts_tablet_manager.cc:1905] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:41:01.119940 28250 ts_tablet_manager.cc:1918] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:41:01.120024 28250 log.cc:1196] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/dfa90bcfddc34c75972cc9398ee18ceb
I20260810 14:41:01.120321 28250 ts_tablet_manager.cc:1939] T dfa90bcfddc34c75972cc9398ee18ceb P bd476e64d6c8463eb74768b9f6b7822f: Deleting consensus metadata
I20260810 14:41:01.121074 27195 catalog_manager.cc:4841] TS bd476e64d6c8463eb74768b9f6b7822f (127.15.84.3:40493): tablet dfa90bcfddc34c75972cc9398ee18ceb (table test-table [id=80f290c727bf4efc8fcb35ddcbd0ecbd]) successfully deleted
I20260810 14:41:01.121186 28253 ts_tablet_manager.cc:1918] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:41:01.121258 28253 log.cc:1196] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/dfa90bcfddc34c75972cc9398ee18ceb
I20260810 14:41:01.121519 28249 ts_tablet_manager.cc:1918] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:41:01.121537 28253 ts_tablet_manager.cc:1939] T dfa90bcfddc34c75972cc9398ee18ceb P f4d041883b7f4dfc8653f1bca560fd9a: Deleting consensus metadata
I20260810 14:41:01.121608 28249 log.cc:1196] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-0/wal/wals/dfa90bcfddc34c75972cc9398ee18ceb
I20260810 14:41:01.121891 28249 ts_tablet_manager.cc:1939] T dfa90bcfddc34c75972cc9398ee18ceb P f48dd5c98ead413e881a9187a8678331: Deleting consensus metadata
I20260810 14:41:01.122397 27195 catalog_manager.cc:4841] TS f4d041883b7f4dfc8653f1bca560fd9a (127.15.84.2:35349): tablet dfa90bcfddc34c75972cc9398ee18ceb (table test-table [id=80f290c727bf4efc8fcb35ddcbd0ecbd]) successfully deleted
I20260810 14:41:01.122478 27444 tablet_service.cc:1467] Processing CreateTablet for tablet 1b52e3da1d244c6fa7c7487700b9ed0a (DEFAULT_TABLE table=test-table [id=531ff1e4b8d34bc593ec664ea07c45f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:41:01.122763 27444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b52e3da1d244c6fa7c7487700b9ed0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:01.122761 27912 tablet_service.cc:1467] Processing CreateTablet for tablet 1b52e3da1d244c6fa7c7487700b9ed0a (DEFAULT_TABLE table=test-table [id=531ff1e4b8d34bc593ec664ea07c45f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:41:01.122923 27663 tablet_service.cc:1467] Processing CreateTablet for tablet 1b52e3da1d244c6fa7c7487700b9ed0a (DEFAULT_TABLE table=test-table [id=531ff1e4b8d34bc593ec664ea07c45f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:41:01.122998 27912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b52e3da1d244c6fa7c7487700b9ed0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:01.123162 27663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b52e3da1d244c6fa7c7487700b9ed0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:01.124960 28078 tablet_bootstrap.cc:492] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331: Bootstrap starting.
I20260810 14:41:01.125011 28079 tablet_bootstrap.cc:492] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f: Bootstrap starting.
I20260810 14:41:01.125833 28078 tablet_bootstrap.cc:654] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:01.125833 28079 tablet_bootstrap.cc:654] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:01.126415 28084 tablet_bootstrap.cc:492] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a: Bootstrap starting.
I20260810 14:41:01.127154 28078 tablet_bootstrap.cc:492] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331: No bootstrap required, opened a new log
I20260810 14:41:01.127215 28084 tablet_bootstrap.cc:654] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:01.127249 28078 ts_tablet_manager.cc:1397] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:41:01.127440 28079 tablet_bootstrap.cc:492] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f: No bootstrap required, opened a new log
I20260810 14:41:01.127534 28079 ts_tablet_manager.cc:1397] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:41:01.128036 28079 raft_consensus.cc:348] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:01.128165 28079 raft_consensus.cc:374] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:01.128207 28079 raft_consensus.cc:729] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd476e64d6c8463eb74768b9f6b7822f, State: Initialized, Role: FOLLOWER
I20260810 14:41:01.128345 28084 tablet_bootstrap.cc:492] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a: No bootstrap required, opened a new log
I20260810 14:41:01.128343 28079 consensus_queue.cc:260] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:01.128428 28084 ts_tablet_manager.cc:1397] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:41:01.128638 28079 ts_tablet_manager.cc:1428] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:41:01.128829 28084 raft_consensus.cc:348] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:01.128942 28084 raft_consensus.cc:374] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:01.128981 28084 raft_consensus.cc:729] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d041883b7f4dfc8653f1bca560fd9a, State: Initialized, Role: FOLLOWER
I20260810 14:41:01.129127 28084 consensus_queue.cc:260] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:01.129415 28084 ts_tablet_manager.cc:1428] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:41:01.129554 28078 raft_consensus.cc:348] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:01.129669 28078 raft_consensus.cc:374] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:01.129707 28078 raft_consensus.cc:729] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48dd5c98ead413e881a9187a8678331, State: Initialized, Role: FOLLOWER
I20260810 14:41:01.129827 28078 consensus_queue.cc:260] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:01.130085 28078 ts_tablet_manager.cc:1428] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260810 14:41:01.130746 27194 catalog_manager.cc:4841] TS f48dd5c98ead413e881a9187a8678331 (127.15.84.1:43271): tablet dfa90bcfddc34c75972cc9398ee18ceb (table test-table [id=80f290c727bf4efc8fcb35ddcbd0ecbd]) successfully deleted
I20260810 14:41:01.133227 28110 raft_consensus.cc:482] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:41:01.133322 28110 raft_consensus.cc:504] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:01.133646 28110 leader_election.cc:288] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f48dd5c98ead413e881a9187a8678331 (127.15.84.1:43271), bd476e64d6c8463eb74768b9f6b7822f (127.15.84.3:40493)
I20260810 14:41:01.134044 27478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b52e3da1d244c6fa7c7487700b9ed0a" candidate_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48dd5c98ead413e881a9187a8678331" is_pre_election: true
I20260810 14:41:01.134052 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b52e3da1d244c6fa7c7487700b9ed0a" candidate_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd476e64d6c8463eb74768b9f6b7822f" is_pre_election: true
I20260810 14:41:01.134182 27478 raft_consensus.cc:2454] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d041883b7f4dfc8653f1bca560fd9a in term 0.
I20260810 14:41:01.134191 27950 raft_consensus.cc:2454] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d041883b7f4dfc8653f1bca560fd9a in term 0.
I20260810 14:41:01.134455 27592 leader_election.cc:302] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f48dd5c98ead413e881a9187a8678331, f4d041883b7f4dfc8653f1bca560fd9a; no voters:
I20260810 14:41:01.134613 28110 raft_consensus.cc:2781] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:41:01.134666 28110 raft_consensus.cc:482] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:41:01.134701 28110 raft_consensus.cc:3037] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:01.135880 28110 raft_consensus.cc:504] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:01.136204 28110 leader_election.cc:288] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [CANDIDATE]: Term 1 election: Requested vote from peers f48dd5c98ead413e881a9187a8678331 (127.15.84.1:43271), bd476e64d6c8463eb74768b9f6b7822f (127.15.84.3:40493)
I20260810 14:41:01.136600 27478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b52e3da1d244c6fa7c7487700b9ed0a" candidate_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48dd5c98ead413e881a9187a8678331"
I20260810 14:41:01.136709 27478 raft_consensus.cc:3037] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:01.137874 27478 raft_consensus.cc:2454] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d041883b7f4dfc8653f1bca560fd9a in term 1.
I20260810 14:41:01.138402 27950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b52e3da1d244c6fa7c7487700b9ed0a" candidate_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd476e64d6c8463eb74768b9f6b7822f"
I20260810 14:41:01.138535 27950 raft_consensus.cc:3037] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:01.139622 27950 raft_consensus.cc:2454] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d041883b7f4dfc8653f1bca560fd9a in term 1.
I20260810 14:41:01.139943 27596 leader_election.cc:302] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd476e64d6c8463eb74768b9f6b7822f, f4d041883b7f4dfc8653f1bca560fd9a; no voters:
I20260810 14:41:01.140219 28110 raft_consensus.cc:2781] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:41:01.140383 28110 raft_consensus.cc:686] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [term 1 LEADER]: Becoming Leader. State: Replica: f4d041883b7f4dfc8653f1bca560fd9a, State: Running, Role: LEADER
I20260810 14:41:01.140520 28110 consensus_queue.cc:237] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } }
I20260810 14:41:01.142295 27210 catalog_manager.cc:5495] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a reported cstate change: term changed from 0 to 1, leader changed from <none> to f4d041883b7f4dfc8653f1bca560fd9a (127.15.84.2). New cstate: current_term: 1 leader_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4d041883b7f4dfc8653f1bca560fd9a" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 } health_report { overall_health: UNKNOWN } } }
I20260810 14:41:01.159204 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:01.156296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.15.84.2:35349)
0810 14:41:01.156630 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:01.156636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:01.156648 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:41:01.156701 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:01.156703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:01.156745 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:01.156815 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:41:01.157003 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:01.157585 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:01.158340 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:01.158551 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:01.158616 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:01.158823 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:41:01.158892 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:41:01.158941 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:01.159007 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 14:41:01.159052 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":176,"threads_started":1}
I20260810 14:41:01.169703 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:01.166390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:48167 (local address 127.15.84.3:40493)
0810 14:41:01.166670 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:01.166679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:01.166695 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:41:01.166818 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:01.166821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:01.166866 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:01.166938 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:41:01.167176 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:01.167780 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:01.168594 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:01.168822 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:01.168900 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:01.169291 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:41:01.169381 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:41:01.169444 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:01.169516 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 14:41:01.169570 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":113,"threads_started":1}
I20260810 14:41:01.169742 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:01.166297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:40493 (local address 127.15.84.2:48167)
0810 14:41:01.166602 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:01.166618 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:01.166735 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:01.166952 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:01.166954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:01.166962 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:41:01.167077 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:01.167082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:01.167867 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:01.167869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:01.168501 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:01.168507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:01.168964 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:01.168967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:01.169044 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:01.169102 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:41:01.169459 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:41:01.169465 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:41:01.169507 (+ 42us) client_negotiation.cc:240] Negotiation successful
0810 14:41:01.169612 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
I20260810 14:41:01.175998 27477 raft_consensus.cc:1261] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Refusing update from remote peer f4d041883b7f4dfc8653f1bca560fd9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:41:01.175997 27950 raft_consensus.cc:1261] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Refusing update from remote peer f4d041883b7f4dfc8653f1bca560fd9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:41:01.176385 28110 consensus_queue.cc:1035] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd476e64d6c8463eb74768b9f6b7822f" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:41:01.176561 28259 consensus_queue.cc:1035] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48dd5c98ead413e881a9187a8678331" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:41:01.192866 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:01.189346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:49617 (local address 127.15.84.2:35349)
0810 14:41:01.189499 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:01.189503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:01.189684 (+ 181us) server_negotiation.cc:407] Connection header received
0810 14:41:01.189795 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:01.189798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:01.189850 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:01.189925 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:41:01.190155 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:01.190659 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:01.191607 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:01.191903 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:01.192014 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:01.192331 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:41:01.192431 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:41:01.192488 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:01.192662 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 14:41:01.192733 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 14:41:01.192873 28282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:01.189292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:35349 (local address 127.15.84.3:49617)
0810 14:41:01.189585 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:01.189601 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:01.189708 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:01.189951 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:01.189954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:01.189963 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:41:01.190077 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:01.190082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:01.190760 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:01.190763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:01.191481 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:01.191492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:01.192054 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:01.192058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:01.192160 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:01.192211 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:41:01.192529 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:41:01.192535 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:41:01.192625 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 14:41:01.192713 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":115,"threads_started":1}
I20260810 14:41:01.213438 27214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:41:01.213591 27214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:41:01.215518 27214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc6e78ae3e3a427cb60b967c0f5c9dde: Sending DeleteTablet for 3 replicas of tablet 1b52e3da1d244c6fa7c7487700b9ed0a
I20260810 14:41:01.216174 27663 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b52e3da1d244c6fa7c7487700b9ed0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:41:01 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:59616
I20260810 14:41:01.216192 27444 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b52e3da1d244c6fa7c7487700b9ed0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:41:01 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:33156
I20260810 14:41:01.216243 27912 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b52e3da1d244c6fa7c7487700b9ed0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:41:01 UTC) from {username='oryx', principal='oryx/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:59050
I20260810 14:41:01.216385 28249 tablet_replica.cc:330] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331: stopping tablet replica
I20260810 14:41:01.216383 28253 tablet_replica.cc:330] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a: stopping tablet replica
I20260810 14:41:01.216451 28250 tablet_replica.cc:330] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f: stopping tablet replica
I20260810 14:41:01.216498 28249 raft_consensus.cc:2229] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:41:01.216497 28253 raft_consensus.cc:2229] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:41:01.216558 28250 raft_consensus.cc:2229] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:41:01.216612 28249 raft_consensus.cc:2258] T 1b52e3da1d244c6fa7c7487700b9ed0a P f48dd5c98ead413e881a9187a8678331 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:41:01.216665 28250 raft_consensus.cc:2258] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:41:01.216699 28253 raft_consensus.cc:2258] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:41:01.216885 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 27337
I20260810 14:41:01.219885 28253 ts_tablet_manager.cc:1905] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:41:01.220279 28250 ts_tablet_manager.cc:1905] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:41:01.223202 28250 ts_tablet_manager.cc:1918] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:41:01.223281 28250 log.cc:1196] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/1b52e3da1d244c6fa7c7487700b9ed0a
I20260810 14:41:01.223512 28250 ts_tablet_manager.cc:1939] T 1b52e3da1d244c6fa7c7487700b9ed0a P bd476e64d6c8463eb74768b9f6b7822f: Deleting consensus metadata
I20260810 14:41:01.223995 28253 ts_tablet_manager.cc:1918] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:41:01.224073 28253 log.cc:1196] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/1b52e3da1d244c6fa7c7487700b9ed0a
I20260810 14:41:01.224315 28253 ts_tablet_manager.cc:1939] T 1b52e3da1d244c6fa7c7487700b9ed0a P f4d041883b7f4dfc8653f1bca560fd9a: Deleting consensus metadata
I20260810 14:41:01.224345 27195 catalog_manager.cc:4841] TS bd476e64d6c8463eb74768b9f6b7822f (127.15.84.3:40493): tablet 1b52e3da1d244c6fa7c7487700b9ed0a (table test-table [id=531ff1e4b8d34bc593ec664ea07c45f2]) successfully deleted
I20260810 14:41:01.226409 27195 catalog_manager.cc:4841] TS f4d041883b7f4dfc8653f1bca560fd9a (127.15.84.2:35349): tablet 1b52e3da1d244c6fa7c7487700b9ed0a (table test-table [id=531ff1e4b8d34bc593ec664ea07c45f2]) successfully deleted
W20260810 14:41:01.236280 27194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:43271 (error 108)
I20260810 14:41:01.237118 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 27566
I20260810 14:41:01.239921 28285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:01.237089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43271 (local address 127.0.0.1:33180)
0810 14:41:01.239711 (+ 2622us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:01.239807 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:43271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2528,"thread_start_us":72,"threads_started":1}
W20260810 14:41:01.240115 27194 catalog_manager.cc:4584] TS f48dd5c98ead413e881a9187a8678331 (127.15.84.1:43271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b52e3da1d244c6fa7c7487700b9ed0a: Network error: Client connection negotiation failed: client connection to 127.15.84.1:43271: connect: Connection refused (error 111)
I20260810 14:41:01.251849 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 27791
I20260810 14:41:01.267061 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 27171
2026-08-10T14:41:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3894 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:41:01 dist-test-slave-rmhg krb5kdc[28290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:41:01 dist-test-slave-rmhg krb5kdc[28290](info): set up 2 sockets
Aug 10 14:41:01 dist-test-slave-rmhg krb5kdc[28290](info): commencing operation
krb5kdc: starting...
W20260810 14:41:02.437381 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:41:02 dist-test-slave-rmhg krb5kdc[28290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:41:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:41:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:41:02.565505 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:37833
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:37833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:41:02.684098 28311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:02.684408 28311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:02.684489 28311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:02.688699 28311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:41:02.688786 28311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:02.688818 28311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:02.688843 28311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:41:02.688862 28311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:41:02.694589 28311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:37833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:37833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:02.695772 28311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:02.696883 28311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:02.706332 28324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:02.706321 28321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:02.706326 28320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:02.706611 28311 server_base.cc:1034] running on GCE node
I20260810 14:41:02.707355 28311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:02.708616 28311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:02.709781 28311 hybrid_clock.cc:648] HybridClock initialized: now 1786372862709764 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T14:41:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1449 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:41:02 dist-test-slave-rmhg krb5kdc[28333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:41:02 dist-test-slave-rmhg krb5kdc[28333](info): set up 2 sockets
Aug 10 14:41:02 dist-test-slave-rmhg krb5kdc[28333](info): commencing operation
krb5kdc: starting...
W20260810 14:41:03.877533 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:41:03 dist-test-slave-rmhg krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:41:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:41:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:41:04.003970 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40167
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:40167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:41:04.119758 28348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:04.120072 28348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:04.120133 28348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:04.124245 28348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:41:04.124325 28348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:04.124354 28348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:04.124375 28348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:41:04.124396 28348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:41:04.130228 28348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:40167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:04.131404 28348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:04.132467 28348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:04.141111 28357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:04.141126 28354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:04.141225 28348 server_base.cc:1034] running on GCE node
W20260810 14:41:04.141126 28355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:04.141955 28348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:04.143095 28348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:04.144266 28348 hybrid_clock.cc:648] HybridClock initialized: now 1786372864144249 us; error 30 us; skew 500 ppm
Aug 10 14:41:04 dist-test-slave-rmhg krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372864, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:41:04.147380 28348 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:41:04.148383 28348 webserver.cc:460] Webserver started at http://127.15.84.62:38611/ using document root <none> and password file <none>
I20260810 14:41:04.148976 28348 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:04.149046 28348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:04.149299 28348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:04.151517 28348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "09494e5acc18b3da97f40480ef236d22"
server_key_iv: "d36674ad8cf3d86d86d197631beb7752"
server_key_version: "encryptionkey@0"
I20260810 14:41:04.152104 28348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "09494e5acc18b3da97f40480ef236d22"
server_key_iv: "d36674ad8cf3d86d86d197631beb7752"
server_key_version: "encryptionkey@0"
I20260810 14:41:04.156116 28348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:41:04.158756 28372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:04.160022 28348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:41:04.160161 28348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "09494e5acc18b3da97f40480ef236d22"
server_key_iv: "d36674ad8cf3d86d86d197631beb7752"
server_key_version: "encryptionkey@0"
I20260810 14:41:04.160272 28348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:04.175874 28348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:04.179474 28348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:04.179680 28348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:04.186924 28348 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:40167
I20260810 14:41:04.186960 28459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:40167 every 8 connection(s)
I20260810 14:41:04.187960 28348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:41:04.191021 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:04.191593 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 28348
I20260810 14:41:04.191748 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:41:04.192032 15696 external_mini_cluster.cc:1428] Setting key 23636470e63299f0bdde2eaac5094708
I20260810 14:41:04.197576 28460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918: Bootstrap starting.
I20260810 14:41:04.200546 28460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918: Neither blocks nor log segments found. Creating new log.
Aug 10 14:41:04 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:41:04.201485 28460 log.cc:824] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918: Log is configured to *not* fsync() on all Append() calls
I20260810 14:41:04.203559 28460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918: No bootstrap required, opened a new log
I20260810 14:41:04.206660 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:04.193434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.15.84.62:40167)
0810 14:41:04.193850 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:04.193867 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:04.193904 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:41:04.194579 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:04.194597 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:04.194947 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:04.195256 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:04.196213 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:04.197301 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:04.198450 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:04.198975 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:04.199101 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:04.201999 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:04.202005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:04.202020 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:41:04.202051 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:04.203718 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:04.204174 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:04.204179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:04.204184 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:04.204306 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:04.204595 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:04.204598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:04.204599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:04.205029 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:04.205247 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:04.205552 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 14:41:04.205871 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260810 14:41:04.207489 28460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40167 } }
I20260810 14:41:04.207789 28460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:04.207850 28460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10fc0aed0f6c42e8a6e9bc7b3e329918, State: Initialized, Role: FOLLOWER
I20260810 14:41:04.208302 28460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40167 } }
I20260810 14:41:04.208431 28460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:41:04.208478 28460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:41:04.208552 28460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:04.209642 28460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40167 } }
I20260810 14:41:04.209964 28460 leader_election.cc:302] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10fc0aed0f6c42e8a6e9bc7b3e329918; no voters:
I20260810 14:41:04.210243 28460 leader_election.cc:288] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:41:04.210368 28469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:41:04.210587 28469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 1 LEADER]: Becoming Leader. State: Replica: 10fc0aed0f6c42e8a6e9bc7b3e329918, State: Running, Role: LEADER
I20260810 14:41:04.210971 28469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40167 } }
I20260810 14:41:04.211469 28460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:41:04.212723 28473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10fc0aed0f6c42e8a6e9bc7b3e329918. Latest consensus state: current_term: 1 leader_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40167 } } }
I20260810 14:41:04.212749 28470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40167 } } }
I20260810 14:41:04.212862 28473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [sys.catalog]: This master's current role is: LEADER
I20260810 14:41:04.212867 28470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [sys.catalog]: This master's current role is: LEADER
I20260810 14:41:04.213260 28483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:41:04.216408 28483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:41:04.221665 28483 catalog_manager.cc:1347] Generated new cluster ID: febd1b8e28904dd7ae187070979e380e
I20260810 14:41:04.221743 28483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:41:04.230620 28483 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:41:04.231616 28483 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:41:04.235553 28483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918: Generated new TSK 0
I20260810 14:41:04.236451 28483 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:41:04.270185 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40167
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:41:04.389917 28505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:04.390225 28505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:04.390270 28505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:04.394395 28505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:04.394466 28505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:04.394529 28505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:41:04.399987 28505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:04.401130 28505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:04.402297 28505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:04.412309 28514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:04.412381 28505 server_base.cc:1034] running on GCE node
W20260810 14:41:04.412274 28521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:04.412271 28517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:04.413039 28505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:04.413784 28505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:04.414943 28505 hybrid_clock.cc:648] HybridClock initialized: now 1786372864414925 us; error 31 us; skew 500 ppm
Aug 10 14:41:04 dist-test-slave-rmhg krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372864, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:41:04.417696 28505 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:41:04.418584 28505 webserver.cc:460] Webserver started at http://127.15.84.1:35299/ using document root <none> and password file <none>
I20260810 14:41:04.419127 28505 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:04.419203 28505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:04.419417 28505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:04.421525 28505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "752f626e903a4cd69578152b925f861b"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "3cbaca0604737bc0f93d26f589ae9e10"
server_key_iv: "e3445a41d918d87923bf434848a78e17"
server_key_version: "encryptionkey@0"
I20260810 14:41:04.422063 28505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "752f626e903a4cd69578152b925f861b"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "3cbaca0604737bc0f93d26f589ae9e10"
server_key_iv: "e3445a41d918d87923bf434848a78e17"
server_key_version: "encryptionkey@0"
I20260810 14:41:04.425873 28505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:41:04.428423 28536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:04.429766 28505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:41:04.429893 28505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "752f626e903a4cd69578152b925f861b"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "3cbaca0604737bc0f93d26f589ae9e10"
server_key_iv: "e3445a41d918d87923bf434848a78e17"
server_key_version: "encryptionkey@0"
I20260810 14:41:04.430002 28505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:04.445078 28505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:04.448312 28505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:04.448509 28505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:04.449136 28505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:04.450106 28505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:04.450156 28505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:04.450225 28505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:04.450253 28505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:04.459348 28505 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:37361
I20260810 14:41:04.459378 28713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:37361 every 8 connection(s)
I20260810 14:41:04.460332 28505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:41:04.467903 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 28505
I20260810 14:41:04.468060 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:41:04.468390 15696 external_mini_cluster.cc:1428] Setting key 1690e02c2e5951ead3170cdfa384b43a
Aug 10 14:41:04 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372864, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:41:04.474241 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:04.462423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:59201 (local address 127.15.84.62:40167)
0810 14:41:04.462561 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:04.462564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:04.463682 (+ 1118us) server_negotiation.cc:407] Connection header received
0810 14:41:04.464706 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:04.464710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:04.464758 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:04.464836 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:04.466171 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:04.466710 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:04.467858 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:04.468205 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:04.468315 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:04.470850 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:04.470853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:04.470855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:41:04.470881 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:04.472150 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:04.472651 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:04.472655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:04.472656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:04.472738 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:04.473069 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:04.473072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:04.473074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:04.473306 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:04.473397 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:04.473927 (+ 530us) server_negotiation.cc:299] Negotiation successful
0810 14:41:04.474064 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:41:04.475106 28716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:04.462838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:59201)
0810 14:41:04.463529 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:04.463575 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:04.464499 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:04.465032 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:04.465040 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:04.465490 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:04.466020 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:04.466035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:04.466809 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:04.466812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:04.467740 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:04.467748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:04.468358 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:04.468361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:04.468545 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:04.469127 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:04.469175 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:04.470683 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:41:04.472265 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:41:04.472271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:41:04.472291 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:41:04.472554 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:41:04.472824 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:41:04.472827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:41:04.472829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:41:04.472979 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:41:04.473409 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:41:04.473416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:41:04.473686 (+ 270us) client_negotiation.cc:769] Sending connection context
0810 14:41:04.473865 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 14:41:04.474219 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":113,"threads_started":1}
I20260810 14:41:04.476393 28714 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40167
I20260810 14:41:04.476649 28714 heartbeater.cc:464] Registering TS with master...
I20260810 14:41:04.477226 28714 heartbeater.cc:511] Master 127.15.84.62:40167 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:41:04.478914 28399 ts_manager.cc:194] Registered new tserver with Master: 752f626e903a4cd69578152b925f861b (127.15.84.1:37361)
I20260810 14:41:04.480115 28399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:59201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:41:04.497493 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40167
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:41:04.615574 28722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:04.615869 28722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:04.615931 28722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:04.620144 28722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:04.620239 28722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:04.620327 28722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:41:04.626744 28722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:04.627885 28722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:04.628931 28722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:04.638952 28733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:04.638943 28731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:04.638943 28735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:04.639009 28722 server_base.cc:1034] running on GCE node
I20260810 14:41:04.639626 28722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:04.640232 28722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:04.641377 28722 hybrid_clock.cc:648] HybridClock initialized: now 1786372864641363 us; error 27 us; skew 500 ppm
Aug 10 14:41:04 dist-test-slave-rmhg krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372864, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:41:04.644168 28722 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:41:04.645236 28722 webserver.cc:460] Webserver started at http://127.15.84.2:42311/ using document root <none> and password file <none>
I20260810 14:41:04.645823 28722 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:04.645889 28722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:04.646102 28722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:04.648273 28722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "f6289967299e4a1286b31b2b8f777f2f"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "1a5a7d525cebf32fc8df526597a835d5"
server_key_iv: "b17ead9b7d1df19bd2cffc6429a86925"
server_key_version: "encryptionkey@0"
I20260810 14:41:04.648847 28722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "f6289967299e4a1286b31b2b8f777f2f"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "1a5a7d525cebf32fc8df526597a835d5"
server_key_iv: "b17ead9b7d1df19bd2cffc6429a86925"
server_key_version: "encryptionkey@0"
I20260810 14:41:04.652719 28722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 14:41:04.655236 28747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:04.656455 28722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:41:04.656579 28722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "f6289967299e4a1286b31b2b8f777f2f"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "1a5a7d525cebf32fc8df526597a835d5"
server_key_iv: "b17ead9b7d1df19bd2cffc6429a86925"
server_key_version: "encryptionkey@0"
I20260810 14:41:04.656690 28722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:04.681367 28722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:04.684938 28722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:04.685192 28722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:04.685879 28722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:04.686821 28722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:04.686873 28722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:04.686929 28722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:04.686959 28722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:04.695994 28722 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:40987
I20260810 14:41:04.696020 28914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:40987 every 8 connection(s)
I20260810 14:41:04.697031 28722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:41:04.705747 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 28722
I20260810 14:41:04.705909 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:41:04.706241 15696 external_mini_cluster.cc:1428] Setting key 3070577876c1d905e2f5784fbd821fff
Aug 10 14:41:04 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372864, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:41:04.711192 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:04.698915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:37961 (local address 127.15.84.62:40167)
0810 14:41:04.699090 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:04.699094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:04.700194 (+ 1100us) server_negotiation.cc:407] Connection header received
0810 14:41:04.701042 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:04.701044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:04.701102 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:04.701212 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:04.702632 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:04.703193 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:04.704412 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:04.704697 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:04.704795 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:04.707549 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:04.707553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:04.707556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:41:04.707582 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:04.708914 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:04.709537 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:04.709540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:04.709541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:04.709617 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:04.710019 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:04.710022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:04.710024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:04.710262 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:04.710355 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:04.710879 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 14:41:04.711026 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 14:41:04.712133 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:04.699354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:37961)
0810 14:41:04.700031 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:04.700076 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:04.700863 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:04.701374 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:04.701381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:04.701794 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:04.702447 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:04.702471 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:04.703315 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:04.703318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:04.704271 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:04.704281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:04.704847 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:04.704850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:04.705011 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:04.705658 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:04.705690 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:04.707405 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:41:04.709039 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:41:04.709046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:41:04.709066 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:41:04.709448 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:41:04.709705 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:41:04.709708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:41:04.709710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:41:04.709917 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:41:04.710356 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:41:04.710361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:41:04.710627 (+ 266us) client_negotiation.cc:769] Sending connection context
0810 14:41:04.710808 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 14:41:04.711091 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":112,"threads_started":1}
I20260810 14:41:04.713488 28915 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40167
I20260810 14:41:04.713785 28915 heartbeater.cc:464] Registering TS with master...
I20260810 14:41:04.714323 28915 heartbeater.cc:511] Master 127.15.84.62:40167 requested a full tablet report, sending...
I20260810 14:41:04.715509 28399 ts_manager.cc:194] Registered new tserver with Master: f6289967299e4a1286b31b2b8f777f2f (127.15.84.2:40987)
I20260810 14:41:04.716310 28399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:37961
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:41:04.736526 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40167
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:41:04.856241 28923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:04.856547 28923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:04.856633 28923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:04.860816 28923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:04.860903 28923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:04.860984 28923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:41:04.866751 28923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:04.867921 28923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:04.868945 28923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:04.878698 28931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:04.878701 28930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:04.878698 28934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:04.879068 28923 server_base.cc:1034] running on GCE node
I20260810 14:41:04.879519 28923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:04.880187 28923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:04.881345 28923 hybrid_clock.cc:648] HybridClock initialized: now 1786372864881322 us; error 36 us; skew 500 ppm
Aug 10 14:41:04 dist-test-slave-rmhg krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372864, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:41:04.884236 28923 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:41:04.885251 28923 webserver.cc:460] Webserver started at http://127.15.84.3:43691/ using document root <none> and password file <none>
I20260810 14:41:04.885824 28923 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:04.885895 28923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:04.886114 28923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:04.888319 28923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "de55882de3804bf1a46f01549b1f3044"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "e5e32f6f01b56422ac4b2f80df672e5f"
server_key_iv: "5f1564c160917879fb55617d69e87b16"
server_key_version: "encryptionkey@0"
I20260810 14:41:04.888914 28923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "de55882de3804bf1a46f01549b1f3044"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "e5e32f6f01b56422ac4b2f80df672e5f"
server_key_iv: "5f1564c160917879fb55617d69e87b16"
server_key_version: "encryptionkey@0"
I20260810 14:41:04.893550 28923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:41:04.896186 28943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:04.897604 28923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:41:04.897727 28923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "de55882de3804bf1a46f01549b1f3044"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "e5e32f6f01b56422ac4b2f80df672e5f"
server_key_iv: "5f1564c160917879fb55617d69e87b16"
server_key_version: "encryptionkey@0"
I20260810 14:41:04.897840 28923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:04.909014 28923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:04.912835 28923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:04.913058 28923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:04.913761 28923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:04.914728 28923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:04.914779 28923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:04.914825 28923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:04.914855 28923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:04.923789 28923 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:35529
I20260810 14:41:04.923810 29112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:35529 every 8 connection(s)
I20260810 14:41:04.924836 28923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:41:04.934352 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 28923
I20260810 14:41:04.934512 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
Aug 10 14:41:04 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372864, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:41:04.934849 15696 external_mini_cluster.cc:1428] Setting key cfc905452b9f4e08866105aaf54d0475
I20260810 14:41:04.939116 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:04.926779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:45751 (local address 127.15.84.62:40167)
0810 14:41:04.926958 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:04.926963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:04.928209 (+ 1246us) server_negotiation.cc:407] Connection header received
0810 14:41:04.929086 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:04.929088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:04.929161 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:04.929251 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:04.930615 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:04.931170 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:04.932392 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:04.932616 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:04.932712 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:04.935492 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:04.935496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:04.935498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:41:04.935527 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:04.937018 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:04.937548 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:04.937551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:04.937553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:04.937640 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:04.937969 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:04.937972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:04.937974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:04.938185 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:04.938297 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:04.938819 (+ 522us) server_negotiation.cc:299] Negotiation successful
0810 14:41:04.938950 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 14:41:04.939970 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:04.927232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:45751)
0810 14:41:04.928033 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:04.928086 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:04.928895 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:04.929481 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:04.929488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:04.929894 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:04.930432 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:04.930447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:04.931310 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:04.931313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:04.932251 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:04.932262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:04.932779 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:04.932784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:04.932984 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:04.933579 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:04.933596 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:04.935301 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:41:04.937125 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:41:04.937131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:41:04.937176 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:41:04.937449 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:41:04.937722 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:41:04.937725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:41:04.937727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:41:04.937880 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:41:04.938287 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:41:04.938294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:41:04.938573 (+ 279us) client_negotiation.cc:769] Sending connection context
0810 14:41:04.938760 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 14:41:04.939118 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":128,"threads_started":1}
I20260810 14:41:04.941045 29115 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40167
I20260810 14:41:04.941308 29115 heartbeater.cc:464] Registering TS with master...
I20260810 14:41:04.941803 29115 heartbeater.cc:511] Master 127.15.84.62:40167 requested a full tablet report, sending...
I20260810 14:41:04.942826 28399 ts_manager.cc:194] Registered new tserver with Master: de55882de3804bf1a46f01549b1f3044 (127.15.84.3:35529)
I20260810 14:41:04.943440 28399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:45751
I20260810 14:41:04.948736 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.1.keytab.
Entry for principal oryx/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.1.keytab.
WARNING: no policy specified for oryx/127.15.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.2.keytab.
Entry for principal oryx/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.2.keytab.
WARNING: no policy specified for oryx/127.15.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.3.keytab.
Entry for principal oryx/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.3.keytab.
I20260810 14:41:05.025693 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 28505
I20260810 14:41:05.045928 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 28722
I20260810 14:41:05.056401 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 28923
I20260810 14:41:05.076254 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 28348
I20260810 14:41:05.086161 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40167
--webserver_interface=127.15.84.62
--webserver_port=38611
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:40167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:41:05.202184 29126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:05.202499 29126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:05.202560 29126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:05.206905 29126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:41:05.206998 29126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:05.207026 29126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:05.207048 29126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:41:05.207067 29126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:41:05.213114 29126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:40167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=38611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:05.214397 29126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:05.215543 29126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:41:05.224831 29126 server_base.cc:1034] running on GCE node
W20260810 14:41:05.224787 29134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:05.224785 29133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:05.224787 29136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:05.225746 29126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:05.226965 29126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:05.228127 29126 hybrid_clock.cc:648] HybridClock initialized: now 1786372865228106 us; error 32 us; skew 500 ppm
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372865, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:41:05.230944 29126 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:41:05.231933 29126 webserver.cc:460] Webserver started at http://127.15.84.62:38611/ using document root <none> and password file <none>
I20260810 14:41:05.232499 29126 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:05.232589 29126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:05.236728 29126 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:41:05.239048 29146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:05.240335 29126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:41:05.240471 29126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "09494e5acc18b3da97f40480ef236d22"
server_key_iv: "d36674ad8cf3d86d86d197631beb7752"
server_key_version: "encryptionkey@0"
I20260810 14:41:05.241027 29126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:05.262377 29126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:05.265810 29126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:05.266001 29126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:05.273362 29126 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:40167
I20260810 14:41:05.273393 29232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:40167 every 8 connection(s)
I20260810 14:41:05.274526 29126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:41:05.279093 29233 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:41:05.280923 29233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918: Bootstrap starting.
I20260810 14:41:05.283537 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 29126
I20260810 14:41:05.283979 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:37361
--local_ip_for_outbound_sockets=127.15.84.1
--tserver_master_addrs=127.15.84.62:40167
--webserver_port=35299
--webserver_interface=127.15.84.1
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.84.1
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:41:05.299049 29233 log.cc:824] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918: Log is configured to *not* fsync() on all Append() calls
I20260810 14:41:05.306432 29233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:41:05.306818 29233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918: Bootstrap complete.
I20260810 14:41:05.311259 29233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40167 } }
I20260810 14:41:05.311717 29233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10fc0aed0f6c42e8a6e9bc7b3e329918, State: Initialized, Role: FOLLOWER
I20260810 14:41:05.312229 29233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40167 } }
I20260810 14:41:05.312359 29233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:41:05.312433 29233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:41:05.312534 29233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:41:05.314720 29233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40167 } }
I20260810 14:41:05.315095 29233 leader_election.cc:302] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10fc0aed0f6c42e8a6e9bc7b3e329918; no voters:
I20260810 14:41:05.315469 29233 leader_election.cc:288] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 14:41:05.315631 29239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:41:05.315886 29239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [term 2 LEADER]: Becoming Leader. State: Replica: 10fc0aed0f6c42e8a6e9bc7b3e329918, State: Running, Role: LEADER
I20260810 14:41:05.316265 29239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40167 } }
I20260810 14:41:05.316761 29233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:41:05.317926 29241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10fc0aed0f6c42e8a6e9bc7b3e329918. Latest consensus state: current_term: 2 leader_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40167 } } }
I20260810 14:41:05.318080 29241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [sys.catalog]: This master's current role is: LEADER
I20260810 14:41:05.318318 29240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fc0aed0f6c42e8a6e9bc7b3e329918" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40167 } } }
I20260810 14:41:05.318413 29240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918 [sys.catalog]: This master's current role is: LEADER
I20260810 14:41:05.318527 29250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:41:05.322276 29250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:41:05.323064 29250 catalog_manager.cc:1259] Loaded cluster ID: febd1b8e28904dd7ae187070979e380e
I20260810 14:41:05.323195 29250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:41:05.325695 29250 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:41:05.326479 29250 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10fc0aed0f6c42e8a6e9bc7b3e329918: Loaded TSK: 0
I20260810 14:41:05.327829 29250 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:41:05.408424 29236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:05.408736 29236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:05.408802 29236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:05.413041 29236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:05.413118 29236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:05.413234 29236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:41:05.418871 29236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.1.keytab
--principal=oryx/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:37361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=35299
--tserver_master_addrs=127.15.84.62:40167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:05.419988 29236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:05.421054 29236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:05.432351 29272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:05.432351 29268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:05.432353 29270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:05.433135 29236 server_base.cc:1034] running on GCE node
I20260810 14:41:05.433642 29236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:05.434360 29236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:05.435524 29236 hybrid_clock.cc:648] HybridClock initialized: now 1786372865435492 us; error 54 us; skew 500 ppm
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372865, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:41:05.438453 29236 init.cc:377] Logged in from keytab as oryx/127.15.84.1@KRBTEST.COM (short username oryx)
I20260810 14:41:05.439478 29236 webserver.cc:460] Webserver started at http://127.15.84.1:35299/ using document root <none> and password file <none>
I20260810 14:41:05.440057 29236 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:05.440141 29236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:05.444516 29236 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:41:05.447003 29283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:05.448472 29236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:41:05.448599 29236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "752f626e903a4cd69578152b925f861b"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "3cbaca0604737bc0f93d26f589ae9e10"
server_key_iv: "e3445a41d918d87923bf434848a78e17"
server_key_version: "encryptionkey@0"
I20260810 14:41:05.449216 29236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:05.470892 29236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:05.474138 29236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:05.474350 29236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:05.474985 29236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:05.475872 29236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:05.475924 29236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:05.475978 29236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:05.476007 29236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:05.485366 29236 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:37361
I20260810 14:41:05.485401 29450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:37361 every 8 connection(s)
I20260810 14:41:05.486482 29236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:41:05.491918 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 29236
I20260810 14:41:05.492453 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:40987
--local_ip_for_outbound_sockets=127.15.84.2
--tserver_master_addrs=127.15.84.62:40167
--webserver_port=42311
--webserver_interface=127.15.84.2
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.84.2
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:05.500451 29455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:05.488850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:49439)
0810 14:41:05.489619 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:05.489674 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:05.490513 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:05.491585 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:05.491593 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:05.492113 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:05.492843 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.492866 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.494378 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.494381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.495289 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.495299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.496300 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.496302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.496444 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.497033 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:05.497049 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:05.499644 (+ 2595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":121,"threads_started":1}
W20260810 14:41:05.500969 29453 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:05.488684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:49439 (local address 127.15.84.62:40167)
0810 14:41:05.489181 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:05.489203 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:05.489793 (+ 590us) server_negotiation.cc:407] Connection header received
0810 14:41:05.490876 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:05.490888 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:05.491200 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:05.491490 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:05.493061 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.494212 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.495424 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.496146 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.496319 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.499977 (+ 3658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:49439: BlockingRecv error: recv got EOF from 127.15.84.1:49439 (error 108)
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":114,"threads_started":1}
W20260810 14:41:05.501108 29451 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:40167 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.84.62:40167: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:05.506987 29455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:05.501713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:38755)
0810 14:41:05.501819 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:05.501833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:05.501920 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:05.502159 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:05.502162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:05.502328 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:05.502430 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.502436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.503242 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.503246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.503862 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.503872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.504310 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.504313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.504399 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.504923 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:05.504934 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:05.506845 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 14:41:05.507109 29453 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:05.501817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:38755 (local address 127.15.84.62:40167)
0810 14:41:05.501948 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:05.501952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:05.501980 (+ 28us) server_negotiation.cc:407] Connection header received
0810 14:41:05.502024 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:05.502026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:05.502067 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:05.502130 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:05.502547 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.503095 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.503984 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.504206 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.504296 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.507007 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:38755: BlockingRecv error: recv got EOF from 127.15.84.1:38755 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:05.512639 29455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:05.507555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:55145)
0810 14:41:05.507700 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:05.507712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:05.507777 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:05.508005 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:05.508007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:05.508128 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:05.508207 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.508211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.509235 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.509237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.509756 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.509764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.510283 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.510286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.510345 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.510813 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:05.510823 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:05.512533 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 14:41:05.512831 29453 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:05.507632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:55145 (local address 127.15.84.62:40167)
0810 14:41:05.507752 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:05.507756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:05.507770 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:41:05.507861 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:05.507864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:05.507911 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:05.507994 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:05.508295 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.509099 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.509850 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.510193 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.510290 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.512713 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:55145: BlockingRecv error: recv got EOF from 127.15.84.1:55145 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 14:41:05.513027 29451 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:41:05.615581 29456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:05.615876 29456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:05.615921 29456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:05.620147 29456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:05.620217 29456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:05.620282 29456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:41:05.626667 29456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.2.keytab
--principal=oryx/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:40987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=42311
--tserver_master_addrs=127.15.84.62:40167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:05.627804 29456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:05.628854 29456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:05.638545 29463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:05.638545 29462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:05.638571 29467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:05.638598 29456 server_base.cc:1034] running on GCE node
I20260810 14:41:05.639250 29456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:05.639932 29456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:05.641121 29456 hybrid_clock.cc:648] HybridClock initialized: now 1786372865641100 us; error 34 us; skew 500 ppm
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372865, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:41:05.643980 29456 init.cc:377] Logged in from keytab as oryx/127.15.84.2@KRBTEST.COM (short username oryx)
I20260810 14:41:05.645030 29456 webserver.cc:460] Webserver started at http://127.15.84.2:42311/ using document root <none> and password file <none>
I20260810 14:41:05.645656 29456 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:05.645741 29456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:05.649899 29456 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:41:05.652310 29479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:05.653767 29456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:41:05.653880 29456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "f6289967299e4a1286b31b2b8f777f2f"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "1a5a7d525cebf32fc8df526597a835d5"
server_key_iv: "b17ead9b7d1df19bd2cffc6429a86925"
server_key_version: "encryptionkey@0"
I20260810 14:41:05.654456 29456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:05.684206 29456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:05.687727 29456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:05.687927 29456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:05.688576 29456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:05.689513 29456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:05.689563 29456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:05.689610 29456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:05.689638 29456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:05.698922 29456 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:40987
I20260810 14:41:05.698947 29660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:40987 every 8 connection(s)
I20260810 14:41:05.700131 29456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:41:05.700927 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 29456
I20260810 14:41:05.701440 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:35529
--local_ip_for_outbound_sockets=127.15.84.3
--tserver_master_addrs=127.15.84.62:40167
--webserver_port=43691
--webserver_interface=127.15.84.3
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.84.3
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 14:41:05.713423 29453 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:05.702292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:47379 (local address 127.15.84.62:40167)
0810 14:41:05.702447 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:05.702452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:05.703844 (+ 1392us) server_negotiation.cc:407] Connection header received
0810 14:41:05.704809 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:05.704812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:05.704868 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:05.704946 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:05.706751 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.707329 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.709033 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.709310 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.709394 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.713286 (+ 3892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:47379: BlockingRecv error: recv got EOF from 127.15.84.2:47379 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:41:05.714138 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:05.702742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:47379)
0810 14:41:05.703507 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:05.703701 (+ 194us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:05.704566 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:05.705213 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:05.705223 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:05.705804 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:05.706558 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.706580 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.707470 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.707474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.708904 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.708916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.709475 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.709480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.709695 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.710337 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:05.710359 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:05.713073 (+ 2714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":130,"threads_started":1}
W20260810 14:41:05.714684 29664 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:40167 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.84.62:40167: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:05.720511 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:05.715185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:51177)
0810 14:41:05.715312 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:05.715328 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:05.715424 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:05.715670 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:05.715673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:05.715836 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:05.715947 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.715952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.716716 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.716719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.717463 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.717476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.718008 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.718012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.718106 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.718579 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:05.718593 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:05.720371 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 14:41:05.720619 29453 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:05.715270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:51177 (local address 127.15.84.62:40167)
0810 14:41:05.715391 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:05.715395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:05.715409 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:41:05.715518 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:05.715521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:05.715566 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:05.715630 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:05.716065 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.716612 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.717632 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.717891 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.717979 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.720517 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:51177: BlockingRecv error: recv got EOF from 127.15.84.2:51177 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:05.726186 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:05.721320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:58941)
0810 14:41:05.721435 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:05.721450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:05.721536 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:05.721753 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:05.721756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:05.721907 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:05.721986 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.721991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.722725 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.722728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.723260 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.723268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.723683 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.723687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.723765 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.724210 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:05.724224 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:05.726036 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 14:41:05.726418 29664 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:41:05.726293 29453 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:05.721393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:58941 (local address 127.15.84.62:40167)
0810 14:41:05.721502 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:05.721506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:05.721518 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:41:05.721614 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:05.721617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:05.721660 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:05.721724 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:05.722092 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.722620 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.723366 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.723579 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.723664 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.726181 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:58941: BlockingRecv error: recv got EOF from 127.15.84.2:58941 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 14:41:05.822105 29668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:05.822374 29668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:05.822422 29668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:05.826505 29668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:05.826574 29668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:05.826645 29668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:41:05.832271 29668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/krb5kdc/oryx_127.15.84.3.keytab
--principal=oryx/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:35529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=43691
--tserver_master_addrs=127.15.84.62:40167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:05.833457 29668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:05.834481 29668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:41:05.844480 29668 server_base.cc:1034] running on GCE node
W20260810 14:41:05.844411 29682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:05.844403 29676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:05.844412 29677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:05.845223 29668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:05.845875 29668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:05.847035 29668 hybrid_clock.cc:648] HybridClock initialized: now 1786372865847019 us; error 30 us; skew 500 ppm
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372865, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:41:05.849792 29668 init.cc:377] Logged in from keytab as oryx/127.15.84.3@KRBTEST.COM (short username oryx)
I20260810 14:41:05.850732 29668 webserver.cc:460] Webserver started at http://127.15.84.3:43691/ using document root <none> and password file <none>
I20260810 14:41:05.851305 29668 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:05.851387 29668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:05.855581 29668 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:41:05.857877 29692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:05.859228 29668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:41:05.859357 29668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "de55882de3804bf1a46f01549b1f3044"
format_stamp: "Formatted at 2026-08-10 14:41:04 on dist-test-slave-rmhg"
server_key: "e5e32f6f01b56422ac4b2f80df672e5f"
server_key_iv: "5f1564c160917879fb55617d69e87b16"
server_key_version: "encryptionkey@0"
I20260810 14:41:05.859933 29668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:05.872483 29668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:05.875429 29668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:05.875592 29668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:05.876173 29668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:05.876989 29668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:05.877038 29668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:05.877075 29668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:05.877095 29668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:05.886152 29668 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:35529
I20260810 14:41:05.886179 29857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:35529 every 8 connection(s)
I20260810 14:41:05.887213 29668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:41:05.890038 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 29668
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 14:41:05.898922 29453 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:05.889177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:36005 (local address 127.15.84.62:40167)
0810 14:41:05.889343 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:05.889347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:05.890616 (+ 1269us) server_negotiation.cc:407] Connection header received
0810 14:41:05.891704 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:05.891706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:05.891740 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:05.891790 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:05.893142 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.893667 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.895090 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.895302 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.895376 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.898818 (+ 3442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:36005: BlockingRecv error: recv got EOF from 127.15.84.3:36005 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:41:05.899765 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:05.889622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:36005)
0810 14:41:05.890450 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:05.890503 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:05.891471 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:05.892001 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:05.892011 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:05.892474 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:05.892997 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.893017 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.893799 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.893803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.894970 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.894982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.895448 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.895452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.895651 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.896131 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:05.896146 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:05.898698 (+ 2552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":118,"threads_started":1}
I20260810 14:41:05.899850 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:05.891101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.15.84.62:40167)
0810 14:41:05.891298 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:05.891302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:05.891313 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:41:05.891353 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:05.891356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:05.891395 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:05.891471 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:05.892379 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.892920 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.893882 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.894103 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.894183 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.895640 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:05.895646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:05.895660 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:41:05.895688 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:05.897539 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:05.897989 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:05.897996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:05.898002 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:05.898166 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:05.898394 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:05.898397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:05.898399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:05.898877 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:05.899073 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:05.899457 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 14:41:05.899701 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":65,"threads_started":1}
W20260810 14:41:05.900286 29858 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:40167 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.84.62:40167: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:05.906047 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:05.900728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:43909)
0810 14:41:05.901272 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:05.901287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:05.901376 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:05.901577 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:05.901579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:05.901729 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:05.901849 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.901857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.902682 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.902686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.903245 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.903255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.903640 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.903643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.903711 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.904213 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:05.904225 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:05.905941 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460}
W20260810 14:41:05.906211 29862 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:05.900804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:43909 (local address 127.15.84.62:40167)
0810 14:41:05.900941 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:05.900944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:05.901341 (+ 397us) server_negotiation.cc:407] Connection header received
0810 14:41:05.901448 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:05.901450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:05.901483 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:05.901541 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:05.901955 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.902574 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.903342 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.903556 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.903630 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.906080 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:43909: BlockingRecv error: recv got EOF from 127.15.84.3:43909 (error 108)
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":82}
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:05 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:05.910923 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:05.906523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:51357)
0810 14:41:05.906634 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:05.906647 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:05.906716 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:05.906887 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:05.906889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:05.907006 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:05.907078 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.907083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.907717 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.907719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.908241 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:05.908248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.908649 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.908651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:05.908711 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.909182 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:05.909197 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:05.910812 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 14:41:05.911051 29862 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:05.906586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:51357 (local address 127.15.84.62:40167)
0810 14:41:05.906722 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:05.906725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:05.906737 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:41:05.906771 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:05.906773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:05.906806 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:05.906869 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:05.907153 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.907637 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.908346 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:05.908563 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:05.908642 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:05.910945 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:51357: BlockingRecv error: recv got EOF from 127.15.84.3:51357 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 14:41:05.911185 29858 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 14:41:06 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:06 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:06.519390 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:06.513788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:57873)
0810 14:41:06.514089 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:06.514104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:06.514205 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:06.514472 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:06.514474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:06.514742 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:06.514851 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:06.514858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:06.515727 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:06.515730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:06.516404 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:06.516414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:06.516887 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:06.516890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:06.516962 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:06.517454 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:06.517470 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:06.519259 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
W20260810 14:41:06.519559 29864 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:06.513932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:57873 (local address 127.15.84.62:40167)
0810 14:41:06.514173 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:06.514176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:06.514191 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:41:06.514295 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:06.514298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:06.514354 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:06.514446 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:06.514956 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:06.515599 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:06.516501 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:06.516769 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:06.516851 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:06.519417 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:57873: BlockingRecv error: recv got EOF from 127.15.84.1:57873 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":96,"threads_started":1}
Aug 10 14:41:06 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:06 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:06.732941 29865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:06.727210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:49627)
0810 14:41:06.727497 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:06.727511 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:06.727623 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:06.727923 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:06.727926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:06.728144 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:06.728296 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:06.728303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:06.729221 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:06.729225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:06.729874 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:06.729884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:06.730339 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:06.730341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:06.730405 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:06.730882 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:06.730896 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:06.732807 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
W20260810 14:41:06.733095 29864 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:06.727321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:49627 (local address 127.15.84.62:40167)
0810 14:41:06.727490 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:06.727495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:06.727590 (+ 95us) server_negotiation.cc:407] Connection header received
0810 14:41:06.727745 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:06.727748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:06.727799 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:06.727865 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:06.728422 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:06.729062 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:06.729999 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:06.730247 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:06.730331 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:06.732965 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:49627: BlockingRecv error: recv got EOF from 127.15.84.2:49627 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 14:41:06 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:06 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:06.917475 29866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:06.911851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:42947)
0810 14:41:06.912113 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:06.912127 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:06.912246 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:06.912575 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:06.912577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:06.912759 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:06.912887 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:06.912893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:06.913725 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:06.913728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:06.914324 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:06.914332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:06.914818 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:06.914820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:06.914889 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:06.915402 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:06.915415 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:06.917293 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
W20260810 14:41:06.917833 29864 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:06.911927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:42947 (local address 127.15.84.62:40167)
0810 14:41:06.912112 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:06.912117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:06.912207 (+ 90us) server_negotiation.cc:407] Connection header received
0810 14:41:06.912407 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:06.912410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:06.912467 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:06.912551 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:06.913025 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:06.913631 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:06.914451 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:06.914719 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:06.914813 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:06.917708 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:42947: BlockingRecv error: recv got EOF from 127.15.84.3:42947 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 10 14:41:07 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:07 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:07.530668 29867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:07.520240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:56759)
0810 14:41:07.520572 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:07.520589 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:07.520710 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:07.520949 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:07.520967 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:07.521199 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:07.521323 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:07.521332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:07.522335 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:07.522339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:07.523000 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:07.523011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:07.523441 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:07.523444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:07.523504 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:07.528436 (+ 4932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:07.528451 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:07.530511 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":98,"threads_started":1}
W20260810 14:41:07.530807 29869 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:07.520384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:56759 (local address 127.15.84.62:40167)
0810 14:41:07.520631 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:07.520635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:07.520653 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:41:07.520795 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:07.520798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:07.520842 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:07.520921 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:07.521452 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:07.522206 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:07.523109 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:07.523350 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:07.523431 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:07.530681 (+ 7250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:56759: BlockingRecv error: recv got EOF from 127.15.84.1:56759 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":101,"threads_started":1}
Aug 10 14:41:07 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:07 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:07.739430 29870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:07.733790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:36835)
0810 14:41:07.734143 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:07.734161 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:07.734297 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:07.734575 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:07.734578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:07.734840 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:07.734980 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:07.734990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:07.735915 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:07.735918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:07.736490 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:07.736498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:07.736894 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:07.736896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:07.736961 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:07.737480 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:07.737496 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:07.739283 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":116,"threads_started":1}
W20260810 14:41:07.739549 29869 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:07.733912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:36835 (local address 127.15.84.62:40167)
0810 14:41:07.734047 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:07.734051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:07.734238 (+ 187us) server_negotiation.cc:407] Connection header received
0810 14:41:07.734385 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:07.734388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:07.734442 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:07.734544 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:07.735104 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:07.735808 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:07.736583 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:07.736812 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:07.736883 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:07.739430 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:36835: BlockingRecv error: recv got EOF from 127.15.84.2:36835 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 14:41:07 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:07 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:07.923760 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:07.918323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:50587)
0810 14:41:07.918610 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:07.918626 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:07.918749 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:07.918995 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:07.918999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:07.919266 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:07.919374 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:07.919381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:07.920161 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:07.920164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:07.920730 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:07.920738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:07.921216 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:07.921220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:07.921308 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:07.921801 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:07.921813 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:07.923572 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
W20260810 14:41:07.923830 29869 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:07.918466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:50587 (local address 127.15.84.62:40167)
0810 14:41:07.918610 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:07.918614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:07.918687 (+ 73us) server_negotiation.cc:407] Connection header received
0810 14:41:07.918838 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:07.918842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:07.918892 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:07.918959 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:07.919492 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:07.920042 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:07.920847 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:07.921071 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:07.921187 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:07.923730 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:50587: BlockingRecv error: recv got EOF from 127.15.84.3:50587 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 14:41:08 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:08 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:08.537263 29874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:08.531670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:56539)
0810 14:41:08.532012 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:08.532028 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:08.532137 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:08.532408 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:08.532410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:08.532594 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:08.532697 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:08.532702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:08.533458 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:08.533462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:08.534031 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:08.534040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:08.534501 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:08.534504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:08.534595 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:08.535118 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:08.535129 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:08.537034 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":103,"threads_started":1}
W20260810 14:41:08.537461 29876 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:08.531837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:56539 (local address 127.15.84.62:40167)
0810 14:41:08.532084 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:08.532088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:08.532104 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:41:08.532245 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:08.532248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:08.532296 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:08.532379 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:08.532792 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:08.533356 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:08.534152 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:08.534388 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:08.534467 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:08.537234 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:56539: BlockingRecv error: recv got EOF from 127.15.84.1:56539 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":68,"threads_started":1}
Aug 10 14:41:08 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:08 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:08.746268 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:08.740277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:48517)
0810 14:41:08.740583 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:08.740599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:08.740709 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:08.740996 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:08.740999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:08.741222 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:08.741338 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:08.741345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:08.742114 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:08.742117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:08.742739 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:08.742749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:08.743151 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:08.743155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:08.743228 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:08.743781 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:08.743793 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:08.746057 (+ 2264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
W20260810 14:41:08.746317 29876 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:08.740352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:48517 (local address 127.15.84.62:40167)
0810 14:41:08.740502 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:08.740505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:08.740671 (+ 166us) server_negotiation.cc:407] Connection header received
0810 14:41:08.740784 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:08.740786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:08.740833 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:08.740904 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:08.741460 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:08.741999 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:08.742834 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:08.743038 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:08.743103 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:08.746197 (+ 3094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:48517: BlockingRecv error: recv got EOF from 127.15.84.2:48517 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 14:41:08 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:08 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:08.930375 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:08.924788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:48175)
0810 14:41:08.925102 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:08.925119 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:08.925265 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:08.925644 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:08.925647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:08.925890 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:08.926025 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:08.926034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:08.926869 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:08.926872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:08.927474 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:08.927483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:08.927886 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:08.927888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:08.927951 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:08.928458 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:08.928471 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:08.930253 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":119,"threads_started":1}
W20260810 14:41:08.930518 29876 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:08.924884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:48175 (local address 127.15.84.62:40167)
0810 14:41:08.925039 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:08.925043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:08.925207 (+ 164us) server_negotiation.cc:407] Connection header received
0810 14:41:08.925366 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:08.925369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:08.925473 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:08.925568 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:08.926150 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:08.926761 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:08.927580 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:08.927797 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:08.927875 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:08.930398 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:48175: BlockingRecv error: recv got EOF from 127.15.84.3:48175 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 14:41:09 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:09 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:09.543581 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:09.538102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:48945)
0810 14:41:09.538436 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:09.538450 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:09.538558 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:09.538800 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:09.538803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:09.538990 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:09.539102 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:09.539107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:09.539821 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:09.539823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:09.540370 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:09.540376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:09.540881 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:09.540885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:09.540979 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:09.541533 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:09.541548 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:09.543456 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":117,"threads_started":1}
W20260810 14:41:09.543740 29881 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:09.538224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:48945 (local address 127.15.84.62:40167)
0810 14:41:09.538468 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:09.538472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:09.538521 (+ 49us) server_negotiation.cc:407] Connection header received
0810 14:41:09.538650 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:09.538652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:09.538699 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:09.538767 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:09.539193 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:09.539740 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:09.540508 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:09.540755 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:09.540862 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:09.543595 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:48945: BlockingRecv error: recv got EOF from 127.15.84.1:48945 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":64,"threads_started":1}
Aug 10 14:41:09 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:09 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:09.752439 29883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:09.747207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:55617)
0810 14:41:09.747488 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:09.747502 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:09.747608 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:09.747828 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:09.747830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:09.748020 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:09.748129 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:09.748134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:09.748853 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:09.748855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:09.749582 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:09.749596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:09.750028 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:09.750030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:09.750092 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:09.750557 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:09.750569 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:09.752311 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":109,"threads_started":1}
W20260810 14:41:09.752601 29881 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:09.747279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:55617 (local address 127.15.84.62:40167)
0810 14:41:09.747432 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:09.747436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:09.747565 (+ 129us) server_negotiation.cc:407] Connection header received
0810 14:41:09.747684 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:09.747686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:09.747734 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:09.747801 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:09.748217 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:09.748762 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:09.749707 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:09.749932 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:09.750020 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:09.752447 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:55617: BlockingRecv error: recv got EOF from 127.15.84.2:55617 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 14:41:09 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:09 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:09.937466 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:09.931373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:44959)
0810 14:41:09.931717 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:09.931735 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:09.931871 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:09.932139 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:09.932142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:09.932355 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:09.932466 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:09.932473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:09.933335 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:09.933339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:09.933955 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:09.933965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:09.934505 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:09.934509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:09.934626 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:09.935256 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:09.935272 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:09.937303 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":122,"threads_started":1}
W20260810 14:41:09.937584 29881 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:09.931580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:44959 (local address 127.15.84.62:40167)
0810 14:41:09.931738 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:09.931742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:09.931815 (+ 73us) server_negotiation.cc:407] Connection header received
0810 14:41:09.931967 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:09.931970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:09.932024 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:09.932098 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:09.932604 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:09.933206 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:09.934090 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:09.934333 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:09.934443 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:09.937475 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:44959: BlockingRecv error: recv got EOF from 127.15.84.3:44959 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 14:41:10 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:10 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:10.550541 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:10.544576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:56805)
0810 14:41:10.544900 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:10.544914 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:10.545025 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:10.545305 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:10.545307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:10.545540 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:10.545702 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:10.545711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:10.546561 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:10.546565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:10.547167 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:10.547178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:10.547706 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:10.547710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:10.547801 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:10.548277 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:10.548288 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:10.550416 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":117,"threads_started":1}
W20260810 14:41:10.550698 29887 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:10.544660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:56805 (local address 127.15.84.62:40167)
0810 14:41:10.544940 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:10.544943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:10.544979 (+ 36us) server_negotiation.cc:407] Connection header received
0810 14:41:10.545112 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:10.545115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:10.545177 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:10.545274 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:10.545838 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:10.546442 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:10.547298 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:10.547579 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:10.547676 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:10.550559 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:56805: BlockingRecv error: recv got EOF from 127.15.84.1:56805 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":69,"threads_started":1}
Aug 10 14:41:10 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:10 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:10.758663 29888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:10.753417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:50707)
0810 14:41:10.753722 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:10.753737 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:10.753836 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:10.754064 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:10.754066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:10.754276 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:10.754381 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:10.754388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:10.755124 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:10.755127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:10.755736 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:10.755746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:10.756171 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:10.756174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:10.756247 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:10.756710 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:10.756721 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:10.758538 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
W20260810 14:41:10.758831 29887 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:10.753555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:50707 (local address 127.15.84.62:40167)
0810 14:41:10.753713 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:10.753716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:10.753799 (+ 83us) server_negotiation.cc:407] Connection header received
0810 14:41:10.753917 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:10.753919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:10.753967 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:10.754038 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:10.754471 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:10.755021 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:10.755851 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:10.756082 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:10.756160 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:10.758701 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:50707: BlockingRecv error: recv got EOF from 127.15.84.2:50707 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 14:41:10 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:10 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:10.944308 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:10.938486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:56367)
0810 14:41:10.938799 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:10.938812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:10.938921 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:10.939223 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:10.939225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:10.939429 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:10.939554 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:10.939563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:10.940498 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:10.940501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:10.941242 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:10.941256 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:10.941713 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:10.941717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:10.941804 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:10.942307 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:10.942318 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:10.944113 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":86,"threads_started":1}
W20260810 14:41:10.944414 29887 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:10.938607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:56367 (local address 127.15.84.62:40167)
0810 14:41:10.938763 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:10.938767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:10.938885 (+ 118us) server_negotiation.cc:407] Connection header received
0810 14:41:10.939000 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:10.939002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:10.939053 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:10.939136 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:10.939696 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:10.940322 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:10.941365 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:10.941594 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:10.941679 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:10.944287 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:56367: BlockingRecv error: recv got EOF from 127.15.84.3:56367 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 14:41:11 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:11 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:11.556840 29891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:11.551332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:56053)
0810 14:41:11.551635 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:11.551650 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:11.551742 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:11.552063 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:11.552065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:11.552327 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:11.552436 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:11.552443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:11.553250 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:11.553252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:11.553883 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:11.553894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:11.554371 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:11.554374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:11.554431 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:11.554908 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:11.554919 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:11.556714 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":100,"threads_started":1}
W20260810 14:41:11.557031 29896 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:11.551434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:56053 (local address 127.15.84.62:40167)
0810 14:41:11.551673 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:11.551676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:11.551693 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:41:11.551867 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:11.551870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:11.551927 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:11.552041 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:11.552552 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:11.553107 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:11.554022 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:11.554270 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:11.554354 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:11.556876 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:56053: BlockingRecv error: recv got EOF from 127.15.84.1:56053 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
Aug 10 14:41:11 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:11 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:11.764842 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:11.759425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:58743)
0810 14:41:11.759734 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:11.759749 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:11.759859 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:11.760127 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:11.760130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:11.760329 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:11.760436 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:11.760442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:11.761207 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:11.761210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:11.761780 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:11.761789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:11.762195 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:11.762198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:11.762268 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:11.762737 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:11.762748 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:11.764687 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":110,"threads_started":1}
W20260810 14:41:11.764978 29896 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:11.759499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:58743 (local address 127.15.84.62:40167)
0810 14:41:11.759627 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:11.759630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:11.759813 (+ 183us) server_negotiation.cc:407] Connection header received
0810 14:41:11.759932 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:11.759934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:11.759980 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:11.760057 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:11.760534 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:11.761074 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:11.761869 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:11.762087 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:11.762162 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:11.764858 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:58743: BlockingRecv error: recv got EOF from 127.15.84.2:58743 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 14:41:11 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:11 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:11.950714 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:11.945226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:44395)
0810 14:41:11.945543 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:11.945556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:11.945680 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:11.945924 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:11.945927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:11.946152 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:11.946303 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:11.946311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:11.947125 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:11.947129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:11.947692 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:11.947722 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:11.948152 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:11.948156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:11.948238 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:11.948746 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:11.948758 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:11.950553 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":117,"threads_started":1}
W20260810 14:41:11.950824 29896 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:11.945327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:44395 (local address 127.15.84.62:40167)
0810 14:41:11.945461 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:11.945465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:11.945635 (+ 170us) server_negotiation.cc:407] Connection header received
0810 14:41:11.945767 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:11.945771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:11.945822 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:11.945888 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:11.946436 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:11.947003 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:11.947827 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:11.948046 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:11.948135 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:11.950721 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:44395: BlockingRecv error: recv got EOF from 127.15.84.3:44395 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 10 14:41:12 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:12 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:12.566553 29901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:12.557753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:54551)
0810 14:41:12.558099 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:12.558114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:12.558235 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:12.558472 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:12.558475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:12.558732 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:12.558845 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:12.558852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:12.559619 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:12.559621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:12.560179 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:12.560186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:12.560905 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:12.560909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:12.560982 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:12.564508 (+ 3526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:12.564523 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:12.566418 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":81,"threads_started":1}
W20260810 14:41:12.566743 29902 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:12.557817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:54551 (local address 127.15.84.62:40167)
0810 14:41:12.558128 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:12.558132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:12.558161 (+ 29us) server_negotiation.cc:407] Connection header received
0810 14:41:12.558318 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:12.558321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:12.558366 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:12.558445 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:12.558966 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:12.559531 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:12.560304 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:12.560550 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:12.560636 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:12.566601 (+ 5965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:54551: BlockingRecv error: recv got EOF from 127.15.84.1:54551 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":114,"threads_started":1}
Aug 10 14:41:12 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:12 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:12.771345 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:12.765739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:46917)
0810 14:41:12.766167 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:12.766182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:12.766304 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:12.766601 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:12.766603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:12.766813 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:12.766953 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:12.766961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:12.767807 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:12.767810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:12.768375 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:12.768384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:12.768791 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:12.768794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:12.768862 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:12.769355 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:12.769366 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:12.771222 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":88,"threads_started":1}
W20260810 14:41:12.771469 29902 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:12.765854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:46917 (local address 127.15.84.62:40167)
0810 14:41:12.766039 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:12.766044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:12.766262 (+ 218us) server_negotiation.cc:407] Connection header received
0810 14:41:12.766416 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:12.766419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:12.766479 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:12.766556 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:12.767067 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:12.767704 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:12.768464 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:12.768694 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:12.768764 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:12.771349 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:46917: BlockingRecv error: recv got EOF from 127.15.84.2:46917 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 14:41:12 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:12 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:12.957718 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:12.951759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:59641)
0810 14:41:12.952116 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:12.952129 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:12.952292 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:12.952555 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:12.952558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:12.952794 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:12.952906 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:12.952913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:12.953745 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:12.953749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:12.954372 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:12.954382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:12.954847 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:12.954851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:12.954935 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:12.955406 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:12.955418 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:12.957551 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":95,"threads_started":1}
W20260810 14:41:12.957773 29902 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:12.951846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:59641 (local address 127.15.84.62:40167)
0810 14:41:12.952030 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:12.952036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:12.952207 (+ 171us) server_negotiation.cc:407] Connection header received
0810 14:41:12.952387 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:12.952391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:12.952445 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:12.952515 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:12.953029 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:12.953632 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:12.954492 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:12.954737 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:12.954828 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:12.957650 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:59641: BlockingRecv error: recv got EOF from 127.15.84.3:59641 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 14:41:13 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:13 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:13.572834 29907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:13.567342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:49031)
0810 14:41:13.567611 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:13.567624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:13.567715 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:13.567939 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:13.567941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:13.568131 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:13.568247 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:13.568252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:13.569005 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:13.569008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:13.569617 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:13.569626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:13.570075 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:13.570079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:13.570164 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:13.570721 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:13.570734 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:13.572709 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
W20260810 14:41:13.573030 29908 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:13.567473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:49031 (local address 127.15.84.62:40167)
0810 14:41:13.567701 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:13.567711 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:13.567725 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:41:13.567796 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:13.567798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:13.567844 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:13.567912 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:13.568338 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:13.568907 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:13.569703 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:13.569940 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:13.570017 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:13.572880 (+ 2863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:49031: BlockingRecv error: recv got EOF from 127.15.84.1:49031 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
Aug 10 14:41:13 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:13 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:13.778386 29910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:13.772304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:36241)
0810 14:41:13.772667 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:13.772682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:13.772843 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:13.773098 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:13.773100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:13.773342 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:13.773452 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:13.773459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:13.774406 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:13.774410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:13.775019 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:13.775029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:13.775514 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:13.775518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:13.775603 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:13.776155 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:13.776168 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:13.778252 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":98,"threads_started":1}
W20260810 14:41:13.778568 29908 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:13.772489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:36241 (local address 127.15.84.62:40167)
0810 14:41:13.772664 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:13.772668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:13.772794 (+ 126us) server_negotiation.cc:407] Connection header received
0810 14:41:13.772936 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:13.772939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:13.772987 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:13.773059 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:13.773585 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:13.774284 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:13.775153 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:13.775393 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:13.775489 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:13.778444 (+ 2955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:36241: BlockingRecv error: recv got EOF from 127.15.84.2:36241 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 14:41:13 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:13 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:13.963579 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:13.958493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:49755)
0810 14:41:13.958780 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:13.958795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:13.958900 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:13.959150 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:13.959152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:13.959339 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:13.959441 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:13.959447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:13.960156 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:13.960159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:13.960720 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:13.960728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:13.961091 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:13.961093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:13.961180 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:13.961620 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:13.961630 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:13.963421 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":94,"threads_started":1}
W20260810 14:41:13.963707 29908 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:13.958558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:49755 (local address 127.15.84.62:40167)
0810 14:41:13.958698 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:13.958703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:13.958870 (+ 167us) server_negotiation.cc:407] Connection header received
0810 14:41:13.958974 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:13.958976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:13.959027 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:13.959123 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:13.959532 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:13.960054 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:13.960807 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:13.961013 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:13.961080 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:13.963577 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:49755: BlockingRecv error: recv got EOF from 127.15.84.3:49755 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 14:41:14 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:14 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:14.579651 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:14.573793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:56853)
0810 14:41:14.574212 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:14.574231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:14.574356 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:14.574656 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:14.574659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:14.574921 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:14.575080 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:14.575091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:14.575871 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:14.575874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:14.576457 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:14.576465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:14.576970 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:14.576973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:14.577045 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:14.577636 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:14.577649 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:14.579525 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":101,"threads_started":1}
W20260810 14:41:14.579813 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:14.573883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:56853 (local address 127.15.84.62:40167)
0810 14:41:14.574168 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:14.574173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:14.574304 (+ 131us) server_negotiation.cc:407] Connection header received
0810 14:41:14.574442 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:14.574445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:14.574501 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:14.574611 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:14.575206 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:14.575779 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:14.576598 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:14.576842 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:14.576937 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:14.579690 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:56853: BlockingRecv error: recv got EOF from 127.15.84.1:56853 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
Aug 10 14:41:14 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:14 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:14.784725 29914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:14.779234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:46609)
0810 14:41:14.779525 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:14.779540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:14.779659 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:14.779926 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:14.779928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:14.780126 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:14.780252 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:14.780258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:14.781001 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:14.781004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:14.781629 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:14.781639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:14.782055 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:14.782057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:14.782122 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:14.782686 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:14.782699 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:14.784592 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
W20260810 14:41:14.784927 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:14.779398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:46609 (local address 127.15.84.62:40167)
0810 14:41:14.779553 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:14.779556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:14.779611 (+ 55us) server_negotiation.cc:407] Connection header received
0810 14:41:14.779752 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:14.779755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:14.779811 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:14.779895 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:14.780356 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:14.780891 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:14.781736 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:14.781965 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:14.782045 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:14.784789 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:46609: BlockingRecv error: recv got EOF from 127.15.84.2:46609 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 14:41:14 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:14 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:14.970091 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:14.964497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:39707)
0810 14:41:14.964775 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:14.964792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:14.964907 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:14.965193 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:14.965197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:14.965457 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:14.965574 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:14.965581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:14.966326 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:14.966328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:14.966895 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:14.966902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:14.967350 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:14.967354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:14.967451 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:14.967968 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:14.967980 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:14.969933 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
W20260810 14:41:14.970199 29913 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:14.964602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:39707 (local address 127.15.84.62:40167)
0810 14:41:14.964758 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:14.964762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:14.964864 (+ 102us) server_negotiation.cc:407] Connection header received
0810 14:41:14.964992 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:14.964994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:14.965046 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:14.965131 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:14.965692 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:14.966236 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:14.967007 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:14.967223 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:14.967306 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:14.970078 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:39707: BlockingRecv error: recv got EOF from 127.15.84.3:39707 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 14:41:15 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:15 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:15.586318 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:15.580591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:59705)
0810 14:41:15.580862 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:15.580876 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:15.580967 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:15.581250 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:15.581252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:15.581446 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:15.581556 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:15.581562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:15.582352 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:15.582356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:15.582955 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:15.582966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:15.583451 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:15.583453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:15.583515 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:15.584065 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:15.584078 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:15.586104 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
W20260810 14:41:15.586406 29919 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:15.580697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:59705 (local address 127.15.84.62:40167)
0810 14:41:15.580948 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:15.580952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:15.580967 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:41:15.581053 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:15.581055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:15.581098 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:15.581210 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:15.581669 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:15.582230 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:15.583103 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:15.583346 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:15.583433 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:15.586284 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:59705: BlockingRecv error: recv got EOF from 127.15.84.1:59705 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
Aug 10 14:41:15 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:15 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:15.791216 29925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:15.785765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:42489)
0810 14:41:15.786069 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:15.786083 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:15.786190 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:15.786449 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:15.786451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:15.786650 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:15.786762 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:15.786769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:15.787509 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:15.787512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:15.788112 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:15.788121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:15.788534 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:15.788538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:15.788609 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:15.789116 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:15.789128 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:15.791022 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":81,"threads_started":1}
W20260810 14:41:15.791324 29919 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:15.785872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:42489 (local address 127.15.84.62:40167)
0810 14:41:15.786021 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:15.786025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:15.786152 (+ 127us) server_negotiation.cc:407] Connection header received
0810 14:41:15.786273 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:15.786275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:15.786321 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:15.786397 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:15.786865 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:15.787397 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:15.788216 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:15.788435 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:15.788511 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:15.791193 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:42489: BlockingRecv error: recv got EOF from 127.15.84.2:42489 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 14:41:15 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:15 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:15.976483 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:15.971109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:48429)
0810 14:41:15.971499 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:15.971513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:15.971613 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:15.971894 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:15.971898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:15.972100 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:15.972228 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:15.972238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:15.973000 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:15.973004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:15.973616 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:15.973626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:15.974068 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:15.974071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:15.974152 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:15.974610 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:15.974622 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:15.976364 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":86,"threads_started":1}
W20260810 14:41:15.976596 29919 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:15.971288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:48429 (local address 127.15.84.62:40167)
0810 14:41:15.971435 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:15.971439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:15.971588 (+ 149us) server_negotiation.cc:407] Connection header received
0810 14:41:15.971712 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:15.971715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:15.971767 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:15.971845 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:15.972349 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:15.972891 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:15.973740 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:15.973964 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:15.974052 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:15.976494 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:48429: BlockingRecv error: recv got EOF from 127.15.84.3:48429 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 14:41:16 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:16 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:16.593051 29929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:16.587379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:59401)
0810 14:41:16.587783 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:16.587799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:16.587909 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:16.588201 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:16.588204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:16.588418 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:16.588550 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:16.588559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:16.589427 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:16.589430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:16.590029 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:16.590038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:16.590440 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:16.590443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:16.590502 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:16.591004 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:16.591016 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:16.592893 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":91,"threads_started":1}
W20260810 14:41:16.593195 29932 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:16.587553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:59401 (local address 127.15.84.62:40167)
0810 14:41:16.587832 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:16.587836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:16.587854 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:41:16.588006 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:16.588009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:16.588056 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:16.588134 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:16.588674 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:16.589333 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:16.590132 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:16.590356 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:16.590431 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:16.593048 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:59401: BlockingRecv error: recv got EOF from 127.15.84.1:59401 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":95,"threads_started":1}
Aug 10 14:41:16 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:16 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:16.797806 29933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:16.792118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:36447)
0810 14:41:16.792431 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:16.792446 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:16.792586 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:16.792905 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:16.792908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:16.793106 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:16.793267 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:16.793276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:16.794101 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:16.794104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:16.794684 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:16.794693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:16.795157 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:16.795160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:16.795232 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:16.795774 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:16.795789 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:16.797664 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":91,"threads_started":1}
W20260810 14:41:16.798105 29932 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:16.792230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:36447 (local address 127.15.84.62:40167)
0810 14:41:16.792369 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:16.792373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:16.792565 (+ 192us) server_negotiation.cc:407] Connection header received
0810 14:41:16.792674 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:16.792677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:16.792733 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:16.792815 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:16.793393 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:16.793985 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:16.794820 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:16.795051 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:16.795130 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:16.797881 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:36447: BlockingRecv error: recv got EOF from 127.15.84.2:36447 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 14:41:16 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:16 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:16.983625 29934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:16.977356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:43579)
0810 14:41:16.977696 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:16.977713 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:16.977847 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:16.978135 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:16.978138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:16.978379 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:16.978526 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:16.978533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:16.979378 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:16.979381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:16.979964 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:16.979974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:16.980505 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:16.980510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:16.980675 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:16.981274 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:16.981287 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:16.983468 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":98,"threads_started":1}
W20260810 14:41:16.983747 29932 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:16.977526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:43579 (local address 127.15.84.62:40167)
0810 14:41:16.977696 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:16.977701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:16.977793 (+ 92us) server_negotiation.cc:407] Connection header received
0810 14:41:16.977943 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:16.977946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:16.978014 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:16.978087 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:16.978671 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:16.979233 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:16.980097 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:16.980343 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:16.980439 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:16.983627 (+ 3188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:43579: BlockingRecv error: recv got EOF from 127.15.84.3:43579 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 14:41:17 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:17 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:17.606739 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:17.593908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:57121)
0810 14:41:17.594212 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:17.594225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:17.594332 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:17.594594 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:17.594596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:17.594789 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:17.594911 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:17.594917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:17.595708 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:17.595710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:17.596287 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:17.596294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:17.596796 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:17.596799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:17.596859 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:17.604218 (+ 7359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:17.604235 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:17.606613 (+ 2378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":89,"threads_started":1}
W20260810 14:41:17.606932 29937 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:17.593967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:57121 (local address 127.15.84.62:40167)
0810 14:41:17.594227 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:17.594231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:17.594281 (+ 50us) server_negotiation.cc:407] Connection header received
0810 14:41:17.594407 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:17.594409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:17.594454 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:17.594568 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:17.595035 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:17.595618 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:17.596422 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:17.596670 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:17.596781 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:17.606785 (+ 10004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:57121: BlockingRecv error: recv got EOF from 127.15.84.1:57121 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
Aug 10 14:41:17 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:17 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:17.803943 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:17.798579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:47577)
0810 14:41:17.798842 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:17.798856 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:17.798960 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:17.799179 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:17.799181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:17.799365 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:17.799470 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:17.799476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:17.800205 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:17.800208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:17.800780 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:17.800788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:17.801289 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:17.801292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:17.801367 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:17.801900 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:17.801913 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:17.803817 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
W20260810 14:41:17.804060 29937 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:17.798634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:47577 (local address 127.15.84.62:40167)
0810 14:41:17.798777 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:17.798782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:17.798923 (+ 141us) server_negotiation.cc:407] Connection header received
0810 14:41:17.799030 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:17.799032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:17.799078 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:17.799151 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:17.799565 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:17.800104 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:17.800889 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:17.801111 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:17.801240 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:17.803959 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:47577: BlockingRecv error: recv got EOF from 127.15.84.2:47577 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 14:41:17 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:17 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:17.990195 29939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:17.984692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:40145)
0810 14:41:17.984986 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:17.984999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:17.985110 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:17.985462 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:17.985465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:17.985669 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:17.985780 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:17.985786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:17.986582 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:17.986585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:17.987153 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:17.987162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:17.987622 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:17.987625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:17.987719 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:17.988182 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:17.988192 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:17.990067 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":101,"threads_started":1}
W20260810 14:41:17.990320 29937 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:17.984812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:40145 (local address 127.15.84.62:40167)
0810 14:41:17.984976 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:17.984979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:17.985057 (+ 78us) server_negotiation.cc:407] Connection header received
0810 14:41:17.985250 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:17.985253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:17.985304 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:17.985389 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:17.985921 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:17.986461 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:17.987276 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:17.987515 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:17.987604 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:17.990217 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:40145: BlockingRecv error: recv got EOF from 127.15.84.3:40145 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 14:41:18 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:18 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:18.613294 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:18.607645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:53675)
0810 14:41:18.607950 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:18.607964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:18.608061 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:18.608294 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:18.608296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:18.608479 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:18.608581 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:18.608587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:18.609334 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:18.609337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:18.609910 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:18.609919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:18.610403 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:18.610406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:18.610466 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:18.611005 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:18.611018 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:18.613098 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":103,"threads_started":1}
W20260810 14:41:18.613449 29943 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:18.607767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:53675 (local address 127.15.84.62:40167)
0810 14:41:18.608051 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:18.608054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:18.608070 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:41:18.608150 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:18.608152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:18.608197 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:18.608266 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:18.608678 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:18.609255 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:18.610055 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:18.610300 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:18.610388 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:18.613308 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:53675: BlockingRecv error: recv got EOF from 127.15.84.1:53675 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":88,"threads_started":1}
Aug 10 14:41:18 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:18 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:18.810536 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:18.804937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:44803)
0810 14:41:18.805256 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:18.805272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:18.805401 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:18.805692 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:18.805695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:18.805954 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:18.806070 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:18.806077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:18.806901 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:18.806905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:18.807484 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:18.807492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:18.807953 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:18.807957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:18.808045 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:18.808560 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:18.808573 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:18.810360 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
W20260810 14:41:18.810635 29943 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:18.805083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:44803 (local address 127.15.84.62:40167)
0810 14:41:18.805254 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:18.805257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:18.805345 (+ 88us) server_negotiation.cc:407] Connection header received
0810 14:41:18.805506 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:18.805509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:18.805566 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:18.805644 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:18.806200 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:18.806783 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:18.807603 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:18.807846 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:18.807936 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:18.810526 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:44803: BlockingRecv error: recv got EOF from 127.15.84.2:44803 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 14:41:18 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:18 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:18.996910 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:18.991141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:45169)
0810 14:41:18.991514 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:18.991529 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:18.991659 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:18.991924 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:18.991927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:18.992143 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:18.992248 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:18.992255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:18.993029 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:18.993033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:18.993674 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:18.993685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:18.994230 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:18.994234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:18.994339 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:18.994915 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:18.994928 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:18.996773 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":117,"threads_started":1}
W20260810 14:41:18.997069 29943 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:18.991293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:45169 (local address 127.15.84.62:40167)
0810 14:41:18.991479 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:18.991484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:18.991612 (+ 128us) server_negotiation.cc:407] Connection header received
0810 14:41:18.991754 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:18.991757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:18.991809 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:18.991879 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:18.992376 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:18.992910 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:18.993807 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:18.994085 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:18.994195 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:18.996927 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:45169: BlockingRecv error: recv got EOF from 127.15.84.3:45169 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 10 14:41:19 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:19 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:19.619693 29946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:19.614257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:37027)
0810 14:41:19.614551 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:19.614565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:19.614659 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:19.614878 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:19.614881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:19.615068 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:19.615174 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:19.615180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:19.615910 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:19.615912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:19.616503 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:19.616511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:19.617003 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:19.617006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:19.617074 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:19.617599 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:19.617611 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:19.619552 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":102,"threads_started":1}
W20260810 14:41:19.619834 29947 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:19.614357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:37027 (local address 127.15.84.62:40167)
0810 14:41:19.614678 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:19.614682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:19.614695 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:41:19.614733 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:19.614736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:19.614777 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:19.614854 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:19.615270 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:19.615824 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:19.616613 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:19.616899 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:19.616980 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:19.619713 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:37027: BlockingRecv error: recv got EOF from 127.15.84.1:37027 (error 108)
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":90,"threads_started":1}
Aug 10 14:41:19 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:19 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:19.817301 29948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:19.811711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:57313)
0810 14:41:19.812038 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:19.812052 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:19.812195 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:19.812470 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:19.812472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:19.812683 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:19.812788 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:19.812795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:19.813617 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:19.813621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:19.814219 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:19.814229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:19.814668 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:19.814672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:19.814755 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:19.815227 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:19.815238 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:19.817082 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":114,"threads_started":1}
W20260810 14:41:19.817586 29947 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:19.811865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:57313 (local address 127.15.84.62:40167)
0810 14:41:19.812010 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:19.812014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:19.812144 (+ 130us) server_negotiation.cc:407] Connection header received
0810 14:41:19.812297 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:19.812301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:19.812350 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:19.812418 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:19.812913 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:19.813506 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:19.814339 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:19.814562 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:19.814657 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:19.817279 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:57313: BlockingRecv error: recv got EOF from 127.15.84.2:57313 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 14:41:19 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:19 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.3@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:20.002950 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:19.997691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.3:33159)
0810 14:41:19.997969 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:19.997983 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:19.998087 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:19.998319 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:19.998322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:19.998509 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:19.998613 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:19.998619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:19.999377 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:19.999379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:19.999944 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:19.999952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:20.000329 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:20.000331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:20.000389 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:20.000891 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:20.000902 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:20.002792 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":90,"threads_started":1}
W20260810 14:41:20.003055 29947 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:19.997777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:33159 (local address 127.15.84.62:40167)
0810 14:41:19.997913 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:19.997916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:19.998047 (+ 131us) server_negotiation.cc:407] Connection header received
0810 14:41:19.998179 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:19.998182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:19.998230 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:19.998287 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:19.998704 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:19.999291 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:20.000029 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:20.000251 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:20.000317 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:20.002950 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.3:33159: BlockingRecv error: recv got EOF from 127.15.84.3:33159 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 14:41:20 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:20 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.1@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:20.625728 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:20.620579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.1:36105)
0810 14:41:20.620841 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:20.620855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:20.620935 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:20.621176 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:20.621179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:20.621367 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:20.621470 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:20.621477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:20.622188 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:20.622191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:20.622780 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:20.622788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:20.623163 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:20.623165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:20.623222 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:20.623698 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:20.623710 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:20.625588 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
W20260810 14:41:20.625901 29959 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:20.620659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:36105 (local address 127.15.84.62:40167)
0810 14:41:20.620910 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:20.620914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:20.620926 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:41:20.621002 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:20.621004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:20.621044 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:20.621120 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:20.621564 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:20.622099 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:20.622868 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:20.623084 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:20.623154 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:20.625769 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.1:36105: BlockingRecv error: recv got EOF from 127.15.84.1:36105 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
Aug 10 14:41:20 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:41:20 dist-test-slave-rmhg krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.84.2@KRBTEST.COM for oryx/127.15.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:41:20.824075 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:20.818393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40167 (local address 127.15.84.2:56057)
0810 14:41:20.818747 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:20.818760 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:20.818875 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:20.819142 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:20.819146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:20.819360 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:20.819464 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:20.819470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:20.820315 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:20.820319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:20.820988 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:20.820999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:20.821503 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:20.821506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:20.821595 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:20.822116 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:20.822127 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:20.823934 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.84.62:40167: Server oryx/127.15.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":113,"threads_started":1}
W20260810 14:41:20.824198 29959 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:20.818472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:56057 (local address 127.15.84.62:40167)
0810 14:41:20.818666 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:20.818671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:20.818840 (+ 169us) server_negotiation.cc:407] Connection header received
0810 14:41:20.818975 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:20.818979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:20.819029 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:20.819098 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:20.819594 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:20.820205 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:20.821103 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:20.821385 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:20.821474 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:20.824086 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.84.2:56057: BlockingRecv error: recv got EOF from 127.15.84.2:56057 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:41:20.891366 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 29236
I20260810 14:41:20.900357 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 29456
I20260810 14:41:20.914295 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 29668
I20260810 14:41:20.930014 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 29126
2026-08-10T14:41:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18214 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T14:41:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:41:20Z Disabled control of system clock
I20260810 14:41:20.971928 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:39083
--webserver_interface=127.15.84.62
--webserver_port=0
--builtin_ntp_servers=127.15.84.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:39083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 14:41:21.090418 29964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:21.090729 29964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:21.090765 29964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:21.094974 29964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:41:21.095059 29964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:21.095073 29964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:21.095090 29964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:41:21.095104 29964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:41:21.100296 29964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:39083
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:39083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:21.101533 29964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:21.103003 29964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:21.112363 29972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:21.112363 29976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:21.112447 29964 server_base.cc:1034] running on GCE node
W20260810 14:41:21.112363 29973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:21.113360 29964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:21.114650 29964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:21.115834 29964 hybrid_clock.cc:648] HybridClock initialized: now 1786372881115815 us; error 39 us; skew 500 ppm
I20260810 14:41:21.118036 29964 webserver.cc:460] Webserver started at http://127.15.84.62:41081/ using document root <none> and password file <none>
I20260810 14:41:21.118680 29964 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:21.118760 29964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:21.118983 29964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:21.121407 29964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "a4fc0800a9c443d0a6c5c16b8cbe354f"
format_stamp: "Formatted at 2026-08-10 14:41:21 on dist-test-slave-rmhg"
server_key: "a5af852c4bc68c571f76c786cec451c9"
server_key_iv: "de1f2701b04526151c5abfe7ec1c6461"
server_key_version: "encryptionkey@0"
I20260810 14:41:21.122021 29964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "a4fc0800a9c443d0a6c5c16b8cbe354f"
format_stamp: "Formatted at 2026-08-10 14:41:21 on dist-test-slave-rmhg"
server_key: "a5af852c4bc68c571f76c786cec451c9"
server_key_iv: "de1f2701b04526151c5abfe7ec1c6461"
server_key_version: "encryptionkey@0"
I20260810 14:41:21.126184 29964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:41:21.128775 29985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:21.130108 29964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:41:21.130236 29964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "a4fc0800a9c443d0a6c5c16b8cbe354f"
format_stamp: "Formatted at 2026-08-10 14:41:21 on dist-test-slave-rmhg"
server_key: "a5af852c4bc68c571f76c786cec451c9"
server_key_iv: "de1f2701b04526151c5abfe7ec1c6461"
server_key_version: "encryptionkey@0"
I20260810 14:41:21.130366 29964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:21.145959 29964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:21.146698 29964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:21.146898 29964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:21.154044 29964 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:39083
I20260810 14:41:21.154064 30067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:39083 every 8 connection(s)
I20260810 14:41:21.155055 29964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:41:21.158418 30070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:21.158948 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 29964
I20260810 14:41:21.159070 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:41:21.159386 15696 external_mini_cluster.cc:1428] Setting key 8f85af0661eca67d355cedace4ee7be3
I20260810 14:41:21.165110 30070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f: Bootstrap starting.
I20260810 14:41:21.167882 30070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:21.168766 30070 log.cc:824] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f: Log is configured to *not* fsync() on all Append() calls
I20260810 14:41:21.169621 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:21.160894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.15.84.62:39083)
0810 14:41:21.161619 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:21.161633 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:21.161670 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:41:21.162351 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:21.162370 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:21.162895 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:21.163349 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:21.163829 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:21.164867 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:21.166126 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:21.166628 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:21.166756 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:21.167620 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:21.167625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:21.167639 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:41:21.168071 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:21.168141 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:41:21.168163 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:21.168258 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:21.168399 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 14:41:21.168687 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":152,"threads_started":1}
I20260810 14:41:21.171106 30070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f: No bootstrap required, opened a new log
I20260810 14:41:21.174793 30070 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fc0800a9c443d0a6c5c16b8cbe354f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39083 } }
I20260810 14:41:21.175042 30070 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:21.175096 30070 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4fc0800a9c443d0a6c5c16b8cbe354f, State: Initialized, Role: FOLLOWER
I20260810 14:41:21.175540 30070 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fc0800a9c443d0a6c5c16b8cbe354f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39083 } }
I20260810 14:41:21.175715 30070 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:41:21.175786 30070 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:41:21.175935 30070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:21.176947 30070 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fc0800a9c443d0a6c5c16b8cbe354f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39083 } }
I20260810 14:41:21.177321 30070 leader_election.cc:302] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4fc0800a9c443d0a6c5c16b8cbe354f; no voters:
I20260810 14:41:21.177637 30070 leader_election.cc:288] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:41:21.177744 30079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:41:21.177973 30079 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [term 1 LEADER]: Becoming Leader. State: Replica: a4fc0800a9c443d0a6c5c16b8cbe354f, State: Running, Role: LEADER
I20260810 14:41:21.178376 30079 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fc0800a9c443d0a6c5c16b8cbe354f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39083 } }
I20260810 14:41:21.178884 30070 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:41:21.180289 30082 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4fc0800a9c443d0a6c5c16b8cbe354f. Latest consensus state: current_term: 1 leader_uuid: "a4fc0800a9c443d0a6c5c16b8cbe354f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fc0800a9c443d0a6c5c16b8cbe354f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39083 } } }
I20260810 14:41:21.180346 30081 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4fc0800a9c443d0a6c5c16b8cbe354f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fc0800a9c443d0a6c5c16b8cbe354f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39083 } } }
I20260810 14:41:21.180531 30081 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [sys.catalog]: This master's current role is: LEADER
I20260810 14:41:21.180531 30082 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f [sys.catalog]: This master's current role is: LEADER
I20260810 14:41:21.180964 30096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:41:21.183980 30096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:41:21.190518 30096 catalog_manager.cc:1347] Generated new cluster ID: fc0f2119d45b4975a575a170dd4c2663
I20260810 14:41:21.190600 30096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:41:21.203308 30096 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:41:21.204315 30096 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:41:21.222456 30096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4fc0800a9c443d0a6c5c16b8cbe354f: Generated new TSK 0
I20260810 14:41:21.223285 30096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:41:21.235931 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39083
--builtin_ntp_servers=127.15.84.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 14:41:21.385756 30113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:21.386039 30113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:21.386080 30113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:21.390127 30113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:21.390196 30113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:21.390251 30113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:41:21.395069 30113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:21.396238 30113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:21.397701 30113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:21.408131 30119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:21.408233 30113 server_base.cc:1034] running on GCE node
W20260810 14:41:21.408170 30122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:21.408131 30120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:21.408856 30113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:21.409653 30113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:21.410856 30113 hybrid_clock.cc:648] HybridClock initialized: now 1786372881410830 us; error 47 us; skew 500 ppm
I20260810 14:41:21.412911 30113 webserver.cc:460] Webserver started at http://127.15.84.1:39145/ using document root <none> and password file <none>
I20260810 14:41:21.413595 30113 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:21.413676 30113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:21.413897 30113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:21.416190 30113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "5df56b05b7d2418080f187d578d97d0e"
format_stamp: "Formatted at 2026-08-10 14:41:21 on dist-test-slave-rmhg"
server_key: "37291ff21c3eb727a06902a7e2940962"
server_key_iv: "5e212c8f085b7b425130e463dcd47f77"
server_key_version: "encryptionkey@0"
I20260810 14:41:21.416837 30113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "5df56b05b7d2418080f187d578d97d0e"
format_stamp: "Formatted at 2026-08-10 14:41:21 on dist-test-slave-rmhg"
server_key: "37291ff21c3eb727a06902a7e2940962"
server_key_iv: "5e212c8f085b7b425130e463dcd47f77"
server_key_version: "encryptionkey@0"
I20260810 14:41:21.420970 30113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 14:41:21.423916 30131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:21.425410 30113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:41:21.425554 30113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "5df56b05b7d2418080f187d578d97d0e"
format_stamp: "Formatted at 2026-08-10 14:41:21 on dist-test-slave-rmhg"
server_key: "37291ff21c3eb727a06902a7e2940962"
server_key_iv: "5e212c8f085b7b425130e463dcd47f77"
server_key_version: "encryptionkey@0"
I20260810 14:41:21.425679 30113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:21.446931 30113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:21.447779 30113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:21.447986 30113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:21.448658 30113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:21.449753 30113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:21.449807 30113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:21.449891 30113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:21.449916 30113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:21.459244 30113 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:40403
I20260810 14:41:21.459277 30314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:40403 every 8 connection(s)
I20260810 14:41:21.460245 30113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:41:21.465250 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 30113
I20260810 14:41:21.465371 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:41:21.465680 15696 external_mini_cluster.cc:1428] Setting key 1d0335d836149d0d8a43288dc8be2348
I20260810 14:41:21.467907 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39083
--builtin_ntp_servers=127.15.84.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 14:41:21.470731 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:21.462236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:39873 (local address 127.15.84.62:39083)
0810 14:41:21.462383 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:21.462387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:21.463545 (+ 1158us) server_negotiation.cc:407] Connection header received
0810 14:41:21.465071 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:21.465074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:21.465139 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:21.465236 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:21.466301 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:21.467022 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:21.468693 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:21.468920 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:21.469004 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:21.470257 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:21.470260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:21.470264 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:41:21.470279 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:21.470304 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:41:21.470314 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:21.470381 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:21.470511 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 14:41:21.470578 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:41:21.471621 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:21.462661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:39083 (local address 127.15.84.1:39873)
0810 14:41:21.463387 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:21.463430 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:21.464822 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:21.465452 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:21.465463 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:21.465563 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:21.466130 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:21.466147 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:21.467185 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:21.467189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:21.468566 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:21.468580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:21.469076 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:21.469079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:21.469328 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:21.469965 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:41:21.469967 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:21.470013 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:41:21.470087 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:41:21.470391 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:41:21.470396 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:41:21.470400 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:41:21.470500 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 14:41:21.470759 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":122,"threads_started":1}
I20260810 14:41:21.472883 30317 heartbeater.cc:347] Connected to a master server at 127.15.84.62:39083
I20260810 14:41:21.473222 30317 heartbeater.cc:464] Registering TS with master...
I20260810 14:41:21.473939 30317 heartbeater.cc:511] Master 127.15.84.62:39083 requested a full tablet report, sending...
I20260810 14:41:21.475746 30018 ts_manager.cc:194] Registered new tserver with Master: 5df56b05b7d2418080f187d578d97d0e (127.15.84.1:40403)
I20260810 14:41:21.476962 30018 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.84.1:39873
W20260810 14:41:21.588202 30323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:21.588531 30323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:21.588572 30323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:21.592890 30323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:21.592978 30323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:21.593034 30323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:41:21.598090 30323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:21.599288 30323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:21.600731 30323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:21.611373 30328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:21.611405 30333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:21.611549 30323 server_base.cc:1034] running on GCE node
W20260810 14:41:21.611747 30330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:21.612219 30323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:21.612939 30323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:21.614145 30323 hybrid_clock.cc:648] HybridClock initialized: now 1786372881614113 us; error 66 us; skew 500 ppm
I20260810 14:41:21.616343 30323 webserver.cc:460] Webserver started at http://127.15.84.2:39123/ using document root <none> and password file <none>
I20260810 14:41:21.616966 30323 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:21.617041 30323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:21.617304 30323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:21.619621 30323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "9beaff4611494afcb1c5742f60f34648"
format_stamp: "Formatted at 2026-08-10 14:41:21 on dist-test-slave-rmhg"
server_key: "19ad6377acebc1a08138205c8913a598"
server_key_iv: "a721e89419ffe8b607d72af61542c2b6"
server_key_version: "encryptionkey@0"
I20260810 14:41:21.620267 30323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "9beaff4611494afcb1c5742f60f34648"
format_stamp: "Formatted at 2026-08-10 14:41:21 on dist-test-slave-rmhg"
server_key: "19ad6377acebc1a08138205c8913a598"
server_key_iv: "a721e89419ffe8b607d72af61542c2b6"
server_key_version: "encryptionkey@0"
I20260810 14:41:21.624511 30323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:41:21.627507 30344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:21.629024 30323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:41:21.629206 30323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "9beaff4611494afcb1c5742f60f34648"
format_stamp: "Formatted at 2026-08-10 14:41:21 on dist-test-slave-rmhg"
server_key: "19ad6377acebc1a08138205c8913a598"
server_key_iv: "a721e89419ffe8b607d72af61542c2b6"
server_key_version: "encryptionkey@0"
I20260810 14:41:21.629341 30323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:21.660346 30323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:21.661253 30323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:21.661468 30323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:21.662175 30323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:21.663273 30323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:21.663341 30323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:21.663403 30323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:21.663429 30323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:21.672824 30323 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:37319
I20260810 14:41:21.672873 30519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:37319 every 8 connection(s)
I20260810 14:41:21.673977 30323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:41:21.676369 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 30323
I20260810 14:41:21.676491 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:41:21.676805 15696 external_mini_cluster.cc:1428] Setting key 3387495d86c1eb8aab120a76a3398fb2
I20260810 14:41:21.679075 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39083
--builtin_ntp_servers=127.15.84.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 14:41:21.686154 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:21.676951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:33109 (local address 127.15.84.62:39083)
0810 14:41:21.677086 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:21.677090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:21.678612 (+ 1522us) server_negotiation.cc:407] Connection header received
0810 14:41:21.680414 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:21.680417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:21.680478 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:21.680554 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:21.681960 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:21.682626 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:21.683892 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:21.684242 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:21.684362 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:21.685586 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:21.685591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:21.685594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:41:21.685614 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:21.685642 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:41:21.685653 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:21.685738 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:21.685892 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 14:41:21.685972 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 14:41:21.687148 30526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:21.677527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:39083 (local address 127.15.84.2:33109)
0810 14:41:21.678463 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:21.678524 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:21.680152 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:21.680804 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:21.680816 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:21.680946 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:21.681761 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:21.681782 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:21.682778 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:21.682781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:21.683746 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:21.683757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:21.684410 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:21.684413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:21.684601 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:21.685278 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:41:21.685279 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:21.685328 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:41:21.685404 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:41:21.685749 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:41:21.685756 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:41:21.685760 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:41:21.685857 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 14:41:21.686144 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":186,"threads_started":1}
I20260810 14:41:21.688508 30524 heartbeater.cc:347] Connected to a master server at 127.15.84.62:39083
I20260810 14:41:21.688799 30524 heartbeater.cc:464] Registering TS with master...
I20260810 14:41:21.689503 30524 heartbeater.cc:511] Master 127.15.84.62:39083 requested a full tablet report, sending...
I20260810 14:41:21.690754 30018 ts_manager.cc:194] Registered new tserver with Master: 9beaff4611494afcb1c5742f60f34648 (127.15.84.2:37319)
I20260810 14:41:21.691596 30018 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.84.2:33109
W20260810 14:41:21.802129 30529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:21.802444 30529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:21.802484 30529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:21.806622 30529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:21.806685 30529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:21.806746 30529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:41:21.811709 30529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:21.812881 30529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:21.814349 30529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:21.824587 30539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:21.824589 30542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:21.824587 30538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:21.825029 30529 server_base.cc:1034] running on GCE node
I20260810 14:41:21.825618 30529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:21.826363 30529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:21.827531 30529 hybrid_clock.cc:648] HybridClock initialized: now 1786372881827499 us; error 55 us; skew 500 ppm
I20260810 14:41:21.829754 30529 webserver.cc:460] Webserver started at http://127.15.84.3:46813/ using document root <none> and password file <none>
I20260810 14:41:21.830410 30529 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:21.830492 30529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:21.830750 30529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:21.833231 30529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "6305aa20592843a784d35affec7a7800"
format_stamp: "Formatted at 2026-08-10 14:41:21 on dist-test-slave-rmhg"
server_key: "b510dc121a1a56d35b7d7c85741527ce"
server_key_iv: "88b48b7eb6e12316b30154d6b97873a7"
server_key_version: "encryptionkey@0"
I20260810 14:41:21.833891 30529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "6305aa20592843a784d35affec7a7800"
format_stamp: "Formatted at 2026-08-10 14:41:21 on dist-test-slave-rmhg"
server_key: "b510dc121a1a56d35b7d7c85741527ce"
server_key_iv: "88b48b7eb6e12316b30154d6b97873a7"
server_key_version: "encryptionkey@0"
I20260810 14:41:21.838236 30529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:41:21.841045 30554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:21.842439 30529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:41:21.842571 30529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "6305aa20592843a784d35affec7a7800"
format_stamp: "Formatted at 2026-08-10 14:41:21 on dist-test-slave-rmhg"
server_key: "b510dc121a1a56d35b7d7c85741527ce"
server_key_iv: "88b48b7eb6e12316b30154d6b97873a7"
server_key_version: "encryptionkey@0"
I20260810 14:41:21.842692 30529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:21.865113 30529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:21.865962 30529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:21.866168 30529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:21.866822 30529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:21.867899 30529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:21.867951 30529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:21.868009 30529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:21.868038 30529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:21.876888 30529 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:39261
I20260810 14:41:21.876915 30733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:39261 every 8 connection(s)
I20260810 14:41:21.877879 30529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:41:21.887615 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:21.879783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:42367 (local address 127.15.84.62:39083)
0810 14:41:21.879937 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:21.879942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:21.881192 (+ 1250us) server_negotiation.cc:407] Connection header received
0810 14:41:21.882715 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:21.882718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:21.882774 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:21.882857 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:21.883860 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:21.884410 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:21.885721 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:21.885962 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:21.886060 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:21.887170 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:21.887173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:21.887175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:41:21.887191 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:21.887212 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:41:21.887220 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:21.887280 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:21.887396 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 14:41:21.887476 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:41:21.887763 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 30529
I20260810 14:41:21.887887 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:41:21.888193 15696 external_mini_cluster.cc:1428] Setting key 9f3af63830307cf9715756af5e3f0de4
I20260810 14:41:21.888867 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:21.880303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:39083 (local address 127.15.84.3:42367)
0810 14:41:21.881042 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:21.881086 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:21.882461 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:21.883029 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:21.883036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:21.883130 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:21.883667 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:21.883681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:21.884553 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:21.884556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:21.885580 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:21.885593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:21.886101 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:21.886103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:21.886261 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:21.886882 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:41:21.886883 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:21.886930 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:41:21.887006 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:41:21.887286 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:41:21.887292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:41:21.887296 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:41:21.887391 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 14:41:21.887745 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":122,"threads_started":1}
I20260810 14:41:21.890195 30734 heartbeater.cc:347] Connected to a master server at 127.15.84.62:39083
I20260810 14:41:21.890457 30734 heartbeater.cc:464] Registering TS with master...
I20260810 14:41:21.891084 30734 heartbeater.cc:511] Master 127.15.84.62:39083 requested a full tablet report, sending...
I20260810 14:41:21.892153 30018 ts_manager.cc:194] Registered new tserver with Master: 6305aa20592843a784d35affec7a7800 (127.15.84.3:39261)
I20260810 14:41:21.892760 30018 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.84.3:42367
I20260810 14:41:21.902808 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 14:41:21.906047 30075 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:21.904993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40842 (local address 127.15.84.62:39083)
0810 14:41:21.905177 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:21.905183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:21.905319 (+ 136us) server_negotiation.cc:407] Connection header received
0810 14:41:21.905453 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:21.905456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:21.905506 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:21.905573 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:21.905915 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40842: BlockingRecv error: recv got EOF from 127.0.0.1:40842 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:41:21.906311 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 30113
I20260810 14:41:21.927479 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 30323
I20260810 14:41:21.953262 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 30529
I20260810 14:41:21.964787 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 29964
2026-08-10T14:41:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1030 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T14:41:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:41:21Z Disabled control of system clock
I20260810 14:41:22.001989 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:36299
--webserver_interface=127.15.84.62
--webserver_port=0
--builtin_ntp_servers=127.15.84.20:34625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:36299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 14:41:22.118817 30756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:22.119122 30756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:22.119159 30756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:22.123245 30756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:41:22.123318 30756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:22.123332 30756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:22.123347 30756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:41:22.123361 30756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:41:22.128365 30756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:36299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:36299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:22.129561 30756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:22.130980 30756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:22.140038 30766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:22.140153 30756 server_base.cc:1034] running on GCE node
W20260810 14:41:22.140084 30763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:22.140041 30769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:22.141032 30756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:22.142315 30756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:22.143507 30756 hybrid_clock.cc:648] HybridClock initialized: now 1786372882143484 us; error 38 us; skew 500 ppm
I20260810 14:41:22.145752 30756 webserver.cc:460] Webserver started at http://127.15.84.62:33969/ using document root <none> and password file <none>
I20260810 14:41:22.146373 30756 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:22.146449 30756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:22.146692 30756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:22.148931 30756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "27a8077b46234f4bb110da0e76a7c586"
format_stamp: "Formatted at 2026-08-10 14:41:22 on dist-test-slave-rmhg"
server_key: "6419b6c88f6527ab7023a2cefdbff3f3"
server_key_iv: "efa3ca998648f71c52d5670253b23bcf"
server_key_version: "encryptionkey@0"
I20260810 14:41:22.149573 30756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "27a8077b46234f4bb110da0e76a7c586"
format_stamp: "Formatted at 2026-08-10 14:41:22 on dist-test-slave-rmhg"
server_key: "6419b6c88f6527ab7023a2cefdbff3f3"
server_key_iv: "efa3ca998648f71c52d5670253b23bcf"
server_key_version: "encryptionkey@0"
I20260810 14:41:22.153659 30756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:41:22.156397 30777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:22.157845 30756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:41:22.157984 30756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "27a8077b46234f4bb110da0e76a7c586"
format_stamp: "Formatted at 2026-08-10 14:41:22 on dist-test-slave-rmhg"
server_key: "6419b6c88f6527ab7023a2cefdbff3f3"
server_key_iv: "efa3ca998648f71c52d5670253b23bcf"
server_key_version: "encryptionkey@0"
I20260810 14:41:22.158102 30756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:22.162977 30756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:22.163646 30756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:22.163818 30756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:22.170965 30756 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:36299
I20260810 14:41:22.170979 30859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:36299 every 8 connection(s)
I20260810 14:41:22.171969 30756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:41:22.175369 30860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:22.178730 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 30756
I20260810 14:41:22.178855 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:41:22.179111 15696 external_mini_cluster.cc:1428] Setting key 4e339ce2a54f0d815a0988e4d795d9d9
I20260810 14:41:22.181529 30860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586: Bootstrap starting.
I20260810 14:41:22.184072 30860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:22.184863 30863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:22.180596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.15.84.62:36299)
0810 14:41:22.181308 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:22.181326 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:22.181363 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:41:22.182031 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:22.182050 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:22.182430 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:22.182711 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:22.183552 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:22.183557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:22.183569 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:41:22.183579 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:22.183639 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:41:22.183661 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:22.183745 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:22.183861 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 14:41:22.184108 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":118,"threads_started":1}
I20260810 14:41:22.184849 30860 log.cc:824] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586: Log is configured to *not* fsync() on all Append() calls
I20260810 14:41:22.187012 30860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586: No bootstrap required, opened a new log
I20260810 14:41:22.190557 30860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a8077b46234f4bb110da0e76a7c586" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36299 } }
I20260810 14:41:22.190814 30860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:22.190873 30860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a8077b46234f4bb110da0e76a7c586, State: Initialized, Role: FOLLOWER
I20260810 14:41:22.191304 30860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a8077b46234f4bb110da0e76a7c586" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36299 } }
I20260810 14:41:22.191437 30860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:41:22.191504 30860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:41:22.191610 30860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:22.192548 30860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a8077b46234f4bb110da0e76a7c586" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36299 } }
I20260810 14:41:22.192893 30860 leader_election.cc:302] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27a8077b46234f4bb110da0e76a7c586; no voters:
I20260810 14:41:22.193239 30860 leader_election.cc:288] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:41:22.193318 30867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:41:22.193578 30867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [term 1 LEADER]: Becoming Leader. State: Replica: 27a8077b46234f4bb110da0e76a7c586, State: Running, Role: LEADER
I20260810 14:41:22.193953 30867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a8077b46234f4bb110da0e76a7c586" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36299 } }
I20260810 14:41:22.194643 30860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:41:22.195781 30868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27a8077b46234f4bb110da0e76a7c586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a8077b46234f4bb110da0e76a7c586" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36299 } } }
I20260810 14:41:22.195804 30870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27a8077b46234f4bb110da0e76a7c586. Latest consensus state: current_term: 1 leader_uuid: "27a8077b46234f4bb110da0e76a7c586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a8077b46234f4bb110da0e76a7c586" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 36299 } } }
I20260810 14:41:22.195927 30868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [sys.catalog]: This master's current role is: LEADER
I20260810 14:41:22.195928 30870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586 [sys.catalog]: This master's current role is: LEADER
I20260810 14:41:22.196341 30882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:41:22.199244 30882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:41:22.204732 30882 catalog_manager.cc:1347] Generated new cluster ID: f1e434c317d54463b050907652f6f3e3
I20260810 14:41:22.204821 30882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:41:22.223085 30882 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:41:22.223431 30882 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:41:22.232189 30882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27a8077b46234f4bb110da0e76a7c586: Generated new TSK 0
I20260810 14:41:22.232933 30882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:41:22.239073 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:36299
--builtin_ntp_servers=127.15.84.20:34625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 14:41:22.378587 30900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:22.378876 30900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:22.378933 30900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:22.384753 30900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:22.384837 30900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:22.384925 30900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:41:22.390511 30900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:22.391739 30900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:22.393239 30900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:41:22.403751 30900 server_base.cc:1034] running on GCE node
W20260810 14:41:22.403658 30907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:22.403643 30913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:22.403658 30909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:22.404398 30900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:22.405087 30900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:22.406273 30900 hybrid_clock.cc:648] HybridClock initialized: now 1786372882406252 us; error 33 us; skew 500 ppm
I20260810 14:41:22.408321 30900 webserver.cc:460] Webserver started at http://127.15.84.1:38035/ using document root <none> and password file <none>
I20260810 14:41:22.408941 30900 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:22.409016 30900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:22.409264 30900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:22.411444 30900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "90cc5f008c884ee280d65f943e0f121a"
format_stamp: "Formatted at 2026-08-10 14:41:22 on dist-test-slave-rmhg"
server_key: "af8799d3173922b2b1ef55a447c9fb06"
server_key_iv: "76968be13f5e17d00974bc964cd5534b"
server_key_version: "encryptionkey@0"
I20260810 14:41:22.412034 30900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "90cc5f008c884ee280d65f943e0f121a"
format_stamp: "Formatted at 2026-08-10 14:41:22 on dist-test-slave-rmhg"
server_key: "af8799d3173922b2b1ef55a447c9fb06"
server_key_iv: "76968be13f5e17d00974bc964cd5534b"
server_key_version: "encryptionkey@0"
I20260810 14:41:22.416294 30900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:41:22.419188 30924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:22.420425 30900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:41:22.420560 30900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "90cc5f008c884ee280d65f943e0f121a"
format_stamp: "Formatted at 2026-08-10 14:41:22 on dist-test-slave-rmhg"
server_key: "af8799d3173922b2b1ef55a447c9fb06"
server_key_iv: "76968be13f5e17d00974bc964cd5534b"
server_key_version: "encryptionkey@0"
I20260810 14:41:22.420673 30900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:22.425303 30900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:22.425937 30900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:22.426096 30900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:22.426640 30900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:22.427556 30900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:22.427604 30900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:22.427652 30900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:22.427682 30900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:22.436455 30900 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:40089
I20260810 14:41:22.436476 31101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:40089 every 8 connection(s)
I20260810 14:41:22.437521 30900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:41:22.438045 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 30900
I20260810 14:41:22.438169 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:41:22.438496 15696 external_mini_cluster.cc:1428] Setting key 85adb3f93d1308989bc57f8e6de3d12c
I20260810 14:41:22.440876 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:36299
--builtin_ntp_servers=127.15.84.20:34625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 14:41:22.444736 30863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:22.440259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:36521 (local address 127.15.84.62:36299)
0810 14:41:22.440388 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:22.440393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:22.441337 (+ 944us) server_negotiation.cc:407] Connection header received
0810 14:41:22.443031 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:22.443034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:22.443083 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:22.443178 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:22.444296 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:22.444299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:22.444301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:41:22.444313 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:22.444334 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:41:22.444342 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:22.444422 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:22.444572 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 14:41:22.444611 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 14:41:22.446002 31107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:22.440187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36299 (local address 127.15.84.1:36521)
0810 14:41:22.441169 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:22.441230 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:22.442779 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:22.443328 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:22.443335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:22.443432 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:22.443977 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:41:22.443978 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:22.444024 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:41:22.444123 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:41:22.444422 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:41:22.444431 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:41:22.444437 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:41:22.444628 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 14:41:22.444924 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":158,"threads_started":1}
I20260810 14:41:22.447428 31104 heartbeater.cc:347] Connected to a master server at 127.15.84.62:36299
I20260810 14:41:22.447677 31104 heartbeater.cc:464] Registering TS with master...
I20260810 14:41:22.448117 31104 heartbeater.cc:511] Master 127.15.84.62:36299 requested a full tablet report, sending...
I20260810 14:41:22.449990 30806 ts_manager.cc:194] Registered new tserver with Master: 90cc5f008c884ee280d65f943e0f121a (127.15.84.1:40089)
W20260810 14:41:22.560554 31110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:22.560863 31110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:22.560904 31110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:22.565075 31110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:22.565171 31110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:22.565233 31110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:41:22.570147 31110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:22.571273 31110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:22.572679 31110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:22.582798 31116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:22.582791 31118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:22.582834 31122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:22.583053 31110 server_base.cc:1034] running on GCE node
I20260810 14:41:22.583644 31110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:22.584409 31110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:22.585683 31110 hybrid_clock.cc:648] HybridClock initialized: now 1786372882585649 us; error 45 us; skew 500 ppm
I20260810 14:41:22.587828 31110 webserver.cc:460] Webserver started at http://127.15.84.2:38537/ using document root <none> and password file <none>
I20260810 14:41:22.588474 31110 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:22.588557 31110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:22.588865 31110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:22.591223 31110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "c6e73d87e11641098fec584dafd8e385"
format_stamp: "Formatted at 2026-08-10 14:41:22 on dist-test-slave-rmhg"
server_key: "543fd5d28c2e22b93aea53624231c767"
server_key_iv: "5f9490667c6417dfb2e87d098214a0a4"
server_key_version: "encryptionkey@0"
I20260810 14:41:22.591960 31110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "c6e73d87e11641098fec584dafd8e385"
format_stamp: "Formatted at 2026-08-10 14:41:22 on dist-test-slave-rmhg"
server_key: "543fd5d28c2e22b93aea53624231c767"
server_key_iv: "5f9490667c6417dfb2e87d098214a0a4"
server_key_version: "encryptionkey@0"
I20260810 14:41:22.596215 31110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:41:22.599292 31130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:22.600768 31110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:41:22.600899 31110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "c6e73d87e11641098fec584dafd8e385"
format_stamp: "Formatted at 2026-08-10 14:41:22 on dist-test-slave-rmhg"
server_key: "543fd5d28c2e22b93aea53624231c767"
server_key_iv: "5f9490667c6417dfb2e87d098214a0a4"
server_key_version: "encryptionkey@0"
I20260810 14:41:22.601024 31110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:22.606179 31110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:22.606920 31110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:22.607108 31110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:22.607754 31110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:22.608830 31110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:22.608884 31110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:22.608942 31110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:22.608970 31110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:22.618323 31110 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:34449
I20260810 14:41:22.618382 31313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:34449 every 8 connection(s)
I20260810 14:41:22.619393 31110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:41:22.629717 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 31110
I20260810 14:41:22.629839 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:41:22.630125 15696 external_mini_cluster.cc:1428] Setting key 7e15fff8a604089310c07948681bed4d
I20260810 14:41:22.631371 30863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:22.621389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:48845 (local address 127.15.84.62:36299)
0810 14:41:22.621527 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:22.621531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:22.622732 (+ 1201us) server_negotiation.cc:407] Connection header received
0810 14:41:22.624133 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:22.624136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:22.624182 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:22.624262 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:22.630857 (+ 6595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:22.630861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:22.630863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:41:22.630879 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:22.630905 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:41:22.630914 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:22.631003 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:22.631176 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 14:41:22.631224 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:41:22.632382 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:22.621841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36299 (local address 127.15.84.2:48845)
0810 14:41:22.622568 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:22.622613 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:22.623888 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:22.624434 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:22.624441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:22.624574 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:22.630436 (+ 5862us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:41:22.630438 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:22.630518 (+ 80us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:41:22.630629 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:41:22.631014 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:41:22.631027 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:41:22.631033 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:41:22.631215 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 14:41:22.631495 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":136,"threads_started":1}
I20260810 14:41:22.632517 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:36299
--builtin_ntp_servers=127.15.84.20:34625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 14:41:22.633786 31318 heartbeater.cc:347] Connected to a master server at 127.15.84.62:36299
I20260810 14:41:22.634083 31318 heartbeater.cc:464] Registering TS with master...
I20260810 14:41:22.634614 31318 heartbeater.cc:511] Master 127.15.84.62:36299 requested a full tablet report, sending...
I20260810 14:41:22.635807 30806 ts_manager.cc:194] Registered new tserver with Master: c6e73d87e11641098fec584dafd8e385 (127.15.84.2:34449)
W20260810 14:41:22.754248 31323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:22.754541 31323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:22.754601 31323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:22.758702 31323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:22.758778 31323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:22.758860 31323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:41:22.764230 31323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:22.765488 31323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:22.767004 31323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:22.777259 31335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:22.777248 31330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:22.777359 31323 server_base.cc:1034] running on GCE node
W20260810 14:41:22.777252 31331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:22.777995 31323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:22.778700 31323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:22.779888 31323 hybrid_clock.cc:648] HybridClock initialized: now 1786372882779852 us; error 51 us; skew 500 ppm
I20260810 14:41:22.781874 31323 webserver.cc:460] Webserver started at http://127.15.84.3:33099/ using document root <none> and password file <none>
I20260810 14:41:22.782459 31323 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:22.782528 31323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:22.782778 31323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:22.784950 31323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "72fe4583d70f4badb7513b4110018c4a"
format_stamp: "Formatted at 2026-08-10 14:41:22 on dist-test-slave-rmhg"
server_key: "e709acbd78a6f2f5cf25ec28ed135f1b"
server_key_iv: "68f88aab6d771e3fa94a25e5148d17ae"
server_key_version: "encryptionkey@0"
I20260810 14:41:22.785557 31323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "72fe4583d70f4badb7513b4110018c4a"
format_stamp: "Formatted at 2026-08-10 14:41:22 on dist-test-slave-rmhg"
server_key: "e709acbd78a6f2f5cf25ec28ed135f1b"
server_key_iv: "68f88aab6d771e3fa94a25e5148d17ae"
server_key_version: "encryptionkey@0"
I20260810 14:41:22.789523 31323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:41:22.792451 31343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:22.793788 31323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:41:22.793919 31323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "72fe4583d70f4badb7513b4110018c4a"
format_stamp: "Formatted at 2026-08-10 14:41:22 on dist-test-slave-rmhg"
server_key: "e709acbd78a6f2f5cf25ec28ed135f1b"
server_key_iv: "68f88aab6d771e3fa94a25e5148d17ae"
server_key_version: "encryptionkey@0"
I20260810 14:41:22.794034 31323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:22.798898 31323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:22.799568 31323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:22.799752 31323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:22.800354 31323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:22.801410 31323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:22.801462 31323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:22.801517 31323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:22.801542 31323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:22.810540 31323 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:38865
I20260810 14:41:22.810563 31517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:38865 every 8 connection(s)
I20260810 14:41:22.811534 31323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:41:22.818032 30863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:22.813495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:60463 (local address 127.15.84.62:36299)
0810 14:41:22.813625 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:22.813628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:22.814951 (+ 1323us) server_negotiation.cc:407] Connection header received
0810 14:41:22.816327 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:22.816330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:22.816374 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:22.816450 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:22.817650 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:22.817653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:22.817655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:41:22.817670 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:22.817691 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:41:22.817699 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:22.817779 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:22.817873 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 14:41:22.817913 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:41:22.819038 31525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:22.813936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:36299 (local address 127.15.84.3:60463)
0810 14:41:22.814779 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:22.814827 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:22.816112 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:22.816629 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:22.816637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:22.816739 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:22.817355 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:41:22.817357 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:22.817403 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:41:22.817485 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:41:22.817766 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:41:22.817771 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:41:22.817775 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:41:22.817915 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 14:41:22.818158 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":140,"threads_started":1}
I20260810 14:41:22.820287 31518 heartbeater.cc:347] Connected to a master server at 127.15.84.62:36299
I20260810 14:41:22.820552 31518 heartbeater.cc:464] Registering TS with master...
I20260810 14:41:22.820737 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 31323
I20260810 14:41:22.820832 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:41:22.821034 31518 heartbeater.cc:511] Master 127.15.84.62:36299 requested a full tablet report, sending...
I20260810 14:41:22.821094 15696 external_mini_cluster.cc:1428] Setting key cd238697528cd8dfe50fc602c7397531
I20260810 14:41:22.822173 30806 ts_manager.cc:194] Registered new tserver with Master: 72fe4583d70f4badb7513b4110018c4a (127.15.84.3:38865)
I20260810 14:41:22.823282 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 14:41:22.826217 30863 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:41:22.825376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.15.84.62:36299)
0810 14:41:22.825522 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:22.825527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:22.825656 (+ 129us) server_negotiation.cc:407] Connection header received
0810 14:41:22.825767 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:22.825769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:22.825800 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:22.825863 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:22.826068 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49324: BlockingRecv error: recv got EOF from 127.0.0.1:49324 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:41:22.826397 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 30900
I20260810 14:41:22.839826 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 31110
I20260810 14:41:22.853042 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 31323
I20260810 14:41:22.863709 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 30756
2026-08-10T14:41:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (899 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:41:22 dist-test-slave-rmhg krb5kdc[31539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:41:22 dist-test-slave-rmhg krb5kdc[31539](info): set up 2 sockets
Aug 10 14:41:22 dist-test-slave-rmhg krb5kdc[31539](info): commencing operation
krb5kdc: starting...
W20260810 14:41:24.010826 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:41:24 dist-test-slave-rmhg krb5kdc[31539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:41:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:41:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:41:24.143610 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:42375
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:42375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:41:24.259656 31557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:24.259917 31557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:24.259974 31557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:24.264005 31557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:41:24.264081 31557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:24.264108 31557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:24.264129 31557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:41:24.264150 31557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:41:24.269523 31557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:42375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:42375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:24.270604 31557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:24.271591 31557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:24.280206 31567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:24.280262 31557 server_base.cc:1034] running on GCE node
W20260810 14:41:24.280200 31569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:24.280206 31572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:24.280989 31557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:24.282150 31557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:24.283329 31557 hybrid_clock.cc:648] HybridClock initialized: now 1786372884283311 us; error 27 us; skew 500 ppm
Aug 10 14:41:24 dist-test-slave-rmhg krb5kdc[31539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372884, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:41:24.286073 31557 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:41:24.286954 31557 webserver.cc:460] Webserver started at http://127.15.84.62:46417/ using document root <none> and password file <none>
I20260810 14:41:24.287492 31557 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:24.287559 31557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:24.287786 31557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:24.289906 31557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "417370ca1a81496683dda9a3a1ce6902"
format_stamp: "Formatted at 2026-08-10 14:41:24 on dist-test-slave-rmhg"
server_key: "f060a23bcc1c48fd6c0077bd5c0791c6"
server_key_iv: "857bd26223f4824fb9184ab560629e18"
server_key_version: "encryptionkey@0"
I20260810 14:41:24.290455 31557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "417370ca1a81496683dda9a3a1ce6902"
format_stamp: "Formatted at 2026-08-10 14:41:24 on dist-test-slave-rmhg"
server_key: "f060a23bcc1c48fd6c0077bd5c0791c6"
server_key_iv: "857bd26223f4824fb9184ab560629e18"
server_key_version: "encryptionkey@0"
I20260810 14:41:24.294250 31557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:41:24.296622 31580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:24.297906 31557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:41:24.298038 31557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "417370ca1a81496683dda9a3a1ce6902"
format_stamp: "Formatted at 2026-08-10 14:41:24 on dist-test-slave-rmhg"
server_key: "f060a23bcc1c48fd6c0077bd5c0791c6"
server_key_iv: "857bd26223f4824fb9184ab560629e18"
server_key_version: "encryptionkey@0"
I20260810 14:41:24.298138 31557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:24.321676 31557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:24.325269 31557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:24.325476 31557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:24.333168 31557 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:42375
I20260810 14:41:24.333431 31659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:42375 every 8 connection(s)
I20260810 14:41:24.334251 31557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:41:24.337141 31660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:24.341012 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 31557
I20260810 14:41:24.341142 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:41:24.341470 15696 external_mini_cluster.cc:1428] Setting key da4a8811e63662d7462a5d97762dbbec
I20260810 14:41:24.343443 31660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902: Bootstrap starting.
I20260810 14:41:24.346027 31660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:24.346800 31660 log.cc:824] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902: Log is configured to *not* fsync() on all Append() calls
I20260810 14:41:24.348984 31660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902: No bootstrap required, opened a new log
Aug 10 14:41:24 dist-test-slave-rmhg krb5kdc[31539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:41:24.352972 31660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417370ca1a81496683dda9a3a1ce6902" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42375 } }
I20260810 14:41:24.353256 31660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:24.353329 31660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417370ca1a81496683dda9a3a1ce6902, State: Initialized, Role: FOLLOWER
I20260810 14:41:24.353802 31660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417370ca1a81496683dda9a3a1ce6902" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42375 } }
I20260810 14:41:24.353942 31660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:41:24.354010 31660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:41:24.354111 31660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:24.355135 31660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417370ca1a81496683dda9a3a1ce6902" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42375 } }
I20260810 14:41:24.355475 31660 leader_election.cc:302] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 417370ca1a81496683dda9a3a1ce6902; no voters:
I20260810 14:41:24.355794 31660 leader_election.cc:288] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:41:24.355847 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:24.342916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.15.84.62:42375)
0810 14:41:24.343395 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:24.343411 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:24.343448 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:41:24.344105 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:24.344124 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:24.344489 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:24.344782 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:24.345679 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:24.346784 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:24.347747 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:24.348246 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:24.348375 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:24.351052 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:24.351057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:24.351072 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:41:24.351101 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:24.352730 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:24.353175 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:24.353182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:24.353188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:24.353347 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:24.353640 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:24.353644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:24.353646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:24.354139 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:24.354349 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:24.354670 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 14:41:24.354986 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":155,"threads_started":1}
I20260810 14:41:24.355921 31666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:41:24.356176 31666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [term 1 LEADER]: Becoming Leader. State: Replica: 417370ca1a81496683dda9a3a1ce6902, State: Running, Role: LEADER
I20260810 14:41:24.356547 31666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417370ca1a81496683dda9a3a1ce6902" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42375 } }
I20260810 14:41:24.357175 31660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:41:24.358232 31668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "417370ca1a81496683dda9a3a1ce6902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417370ca1a81496683dda9a3a1ce6902" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42375 } } }
I20260810 14:41:24.358354 31668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [sys.catalog]: This master's current role is: LEADER
I20260810 14:41:24.358695 31667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 417370ca1a81496683dda9a3a1ce6902. Latest consensus state: current_term: 1 leader_uuid: "417370ca1a81496683dda9a3a1ce6902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417370ca1a81496683dda9a3a1ce6902" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42375 } } }
I20260810 14:41:24.358779 31667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902 [sys.catalog]: This master's current role is: LEADER
I20260810 14:41:24.359125 31677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:41:24.362453 31677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:41:24.368717 31677 catalog_manager.cc:1347] Generated new cluster ID: 2d73a97f9eda473d9c224696bf465d1c
I20260810 14:41:24.368795 31677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:41:24.381769 31677 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:41:24.383330 31677 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:41:24.392298 31677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902: Generated new TSK 0
I20260810 14:41:24.393366 31677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:41:24.431844 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:42375
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:41:24.558362 31697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:24.558643 31697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:24.558687 31697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:24.562719 31697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:24.562783 31697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:24.562842 31697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:41:24.568240 31697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:42375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:24.569345 31697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:24.570479 31697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:24.580885 31705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:24.580885 31710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:24.580888 31706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:24.581190 31697 server_base.cc:1034] running on GCE node
I20260810 14:41:24.581741 31697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:24.582446 31697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:24.583673 31697 hybrid_clock.cc:648] HybridClock initialized: now 1786372884583643 us; error 43 us; skew 500 ppm
Aug 10 14:41:24 dist-test-slave-rmhg krb5kdc[31539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372884, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:41:24.586714 31697 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:41:24.587685 31697 webserver.cc:460] Webserver started at http://127.15.84.1:34343/ using document root <none> and password file <none>
I20260810 14:41:24.588261 31697 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:24.588332 31697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:24.588532 31697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:24.590801 31697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "cf166d73d05a455f8a7072f198d00a2a"
format_stamp: "Formatted at 2026-08-10 14:41:24 on dist-test-slave-rmhg"
server_key: "e20ad9a494a9e6168a8316417dea73f2"
server_key_iv: "b1d02ef50aa01131aefa49cf3fa85ca6"
server_key_version: "encryptionkey@0"
I20260810 14:41:24.591444 31697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "cf166d73d05a455f8a7072f198d00a2a"
format_stamp: "Formatted at 2026-08-10 14:41:24 on dist-test-slave-rmhg"
server_key: "e20ad9a494a9e6168a8316417dea73f2"
server_key_iv: "b1d02ef50aa01131aefa49cf3fa85ca6"
server_key_version: "encryptionkey@0"
I20260810 14:41:24.595521 31697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:41:24.598258 31718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:24.599632 31697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:41:24.599762 31697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "cf166d73d05a455f8a7072f198d00a2a"
format_stamp: "Formatted at 2026-08-10 14:41:24 on dist-test-slave-rmhg"
server_key: "e20ad9a494a9e6168a8316417dea73f2"
server_key_iv: "b1d02ef50aa01131aefa49cf3fa85ca6"
server_key_version: "encryptionkey@0"
I20260810 14:41:24.599872 31697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:24.632706 31697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:24.636078 31697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:24.636277 31697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:24.636917 31697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:24.637833 31697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:24.637883 31697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:24.637938 31697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:24.637964 31697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:24.647080 31697 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:41745
I20260810 14:41:24.647113 31895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:41745 every 8 connection(s)
I20260810 14:41:24.648053 31697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:41:24.650940 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 31697
I20260810 14:41:24.651058 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:41:24.651377 15696 external_mini_cluster.cc:1428] Setting key c820f38ebe83cc3ca0a93c6b57c059d8
Aug 10 14:41:24 dist-test-slave-rmhg krb5kdc[31539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372884, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:41:24.664008 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:24.650032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:52547 (local address 127.15.84.62:42375)
0810 14:41:24.650198 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:24.650201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:24.651147 (+ 946us) server_negotiation.cc:407] Connection header received
0810 14:41:24.652871 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:24.652874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:24.652922 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:24.653008 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:24.654341 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:24.655087 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:24.656813 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:24.657126 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:24.657265 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:24.660284 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:24.660288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:24.660291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:41:24.660320 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:24.661721 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:24.662397 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:24.662399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:24.662400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:24.662471 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:24.662856 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:24.662858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:24.662860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:24.663055 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:24.663136 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:24.663734 (+ 598us) server_negotiation.cc:299] Negotiation successful
0810 14:41:24.663872 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:41:24.664912 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:24.650442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:42375 (local address 127.15.84.1:52547)
0810 14:41:24.650963 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:24.651033 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:24.652612 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:24.653231 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:24.653240 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:24.653642 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:24.654178 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:24.654192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:24.655269 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:24.655273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:24.656680 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:24.656694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:24.657307 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:24.657310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:24.657563 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:24.658263 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:24.658282 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:24.660078 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:41:24.661863 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:41:24.661871 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:41:24.661899 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:41:24.662301 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:41:24.662553 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:41:24.662556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:41:24.662558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:41:24.662759 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:41:24.663171 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:41:24.663177 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:41:24.663467 (+ 290us) client_negotiation.cc:769] Sending connection context
0810 14:41:24.663653 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 14:41:24.663970 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
I20260810 14:41:24.666335 31896 heartbeater.cc:347] Connected to a master server at 127.15.84.62:42375
I20260810 14:41:24.666615 31896 heartbeater.cc:464] Registering TS with master...
I20260810 14:41:24.667239 31896 heartbeater.cc:511] Master 127.15.84.62:42375 requested a full tablet report, sending...
I20260810 14:41:24.669134 31607 ts_manager.cc:194] Registered new tserver with Master: cf166d73d05a455f8a7072f198d00a2a (127.15.84.1:41745)
I20260810 14:41:24.670717 31607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:52547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:41:24.682279 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:42375
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:41:24.800546 31905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:24.800848 31905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:24.800897 31905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:24.805120 31905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:24.805243 31905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:24.805320 31905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:41:24.810967 31905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:42375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:24.812161 31905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:24.813302 31905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:24.823767 31914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:24.823884 31905 server_base.cc:1034] running on GCE node
W20260810 14:41:24.823752 31916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:24.823757 31918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:24.824644 31905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:24.825412 31905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:24.826587 31905 hybrid_clock.cc:648] HybridClock initialized: now 1786372884826549 us; error 52 us; skew 500 ppm
Aug 10 14:41:24 dist-test-slave-rmhg krb5kdc[31539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372884, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:41:24.829649 31905 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:41:24.830650 31905 webserver.cc:460] Webserver started at http://127.15.84.2:38767/ using document root <none> and password file <none>
I20260810 14:41:24.831220 31905 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:24.831284 31905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:24.831473 31905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:24.833752 31905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "08e1a418ce3a4394a3f1b3e081eca2a8"
format_stamp: "Formatted at 2026-08-10 14:41:24 on dist-test-slave-rmhg"
server_key: "2e3fc5700f10e989c1f21dab442f6b42"
server_key_iv: "6e651dcc244afd5fb2e8ad4b8980d152"
server_key_version: "encryptionkey@0"
I20260810 14:41:24.834338 31905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "08e1a418ce3a4394a3f1b3e081eca2a8"
format_stamp: "Formatted at 2026-08-10 14:41:24 on dist-test-slave-rmhg"
server_key: "2e3fc5700f10e989c1f21dab442f6b42"
server_key_iv: "6e651dcc244afd5fb2e8ad4b8980d152"
server_key_version: "encryptionkey@0"
I20260810 14:41:24.838459 31905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 14:41:24.841378 31926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:24.842758 31905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:41:24.842890 31905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "08e1a418ce3a4394a3f1b3e081eca2a8"
format_stamp: "Formatted at 2026-08-10 14:41:24 on dist-test-slave-rmhg"
server_key: "2e3fc5700f10e989c1f21dab442f6b42"
server_key_iv: "6e651dcc244afd5fb2e8ad4b8980d152"
server_key_version: "encryptionkey@0"
I20260810 14:41:24.843005 31905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:24.855875 31905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:24.859789 31905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:24.860014 31905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:24.860661 31905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:24.861670 31905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:24.861723 31905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:24.861774 31905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:24.861804 31905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:24.871188 31905 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:46361
I20260810 14:41:24.871213 32110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:46361 every 8 connection(s)
I20260810 14:41:24.872156 31905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:41:24.880535 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 31905
I20260810 14:41:24.880646 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:41:24.880896 15696 external_mini_cluster.cc:1428] Setting key 0415ef5a253ac3a3ebd837816e054168
Aug 10 14:41:24 dist-test-slave-rmhg krb5kdc[31539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372884, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:41:24.886469 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:24.874112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:49513 (local address 127.15.84.62:42375)
0810 14:41:24.874238 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:24.874242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:24.875179 (+ 937us) server_negotiation.cc:407] Connection header received
0810 14:41:24.876599 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:24.876603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:24.876681 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:24.876755 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:24.878163 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:24.878705 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:24.879900 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:24.880125 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:24.880219 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:24.882865 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:24.882868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:24.882870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:41:24.882897 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:24.884213 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:24.884766 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:24.884769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:24.884770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:24.884849 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:24.885250 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:24.885254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:24.885256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:24.885454 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:24.885546 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:24.886210 (+ 664us) server_negotiation.cc:299] Negotiation successful
0810 14:41:24.886341 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:41:24.887655 32114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:24.874515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:42375 (local address 127.15.84.2:49513)
0810 14:41:24.875017 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:24.875066 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:24.876372 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:24.876989 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:24.876999 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:24.877469 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:24.878004 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:24.878020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:24.878833 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:24.878837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:24.879768 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:24.879779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:24.880285 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:24.880289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:24.880474 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:24.881077 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:24.881096 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:24.882694 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:41:24.884341 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:41:24.884348 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:41:24.884368 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:41:24.884664 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:41:24.884937 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:41:24.884940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:41:24.884942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:41:24.885106 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:41:24.885563 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:41:24.885571 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:41:24.885909 (+ 338us) client_negotiation.cc:769] Sending connection context
0810 14:41:24.886122 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 14:41:24.886510 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":121,"threads_started":1}
I20260810 14:41:24.888972 32111 heartbeater.cc:347] Connected to a master server at 127.15.84.62:42375
I20260810 14:41:24.889310 32111 heartbeater.cc:464] Registering TS with master...
I20260810 14:41:24.889991 32111 heartbeater.cc:511] Master 127.15.84.62:42375 requested a full tablet report, sending...
I20260810 14:41:24.891268 31607 ts_manager.cc:194] Registered new tserver with Master: 08e1a418ce3a4394a3f1b3e081eca2a8 (127.15.84.2:46361)
I20260810 14:41:24.891887 31607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:49513
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:41:24.912402 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:42375
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:41:25.029042 32117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:41:25.029421 32117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:41:25.029492 32117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:41:25.033600 32117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:41:25.033685 32117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:41:25.033766 32117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:41:25.039311 32117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:42375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:41:25.040510 32117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:41:25.041639 32117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:41:25.051515 32125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:41:25.051517 32131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:25.051573 32117 server_base.cc:1034] running on GCE node
W20260810 14:41:25.051517 32126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:41:25.052193 32117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:41:25.052877 32117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:41:25.054044 32117 hybrid_clock.cc:648] HybridClock initialized: now 1786372885054004 us; error 51 us; skew 500 ppm
Aug 10 14:41:25 dist-test-slave-rmhg krb5kdc[31539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372885, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:41:25.056895 32117 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:41:25.058102 32117 webserver.cc:460] Webserver started at http://127.15.84.3:46413/ using document root <none> and password file <none>
I20260810 14:41:25.058677 32117 fs_manager.cc:359] Metadata directory not provided
I20260810 14:41:25.058779 32117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:41:25.059024 32117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:41:25.061220 32117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee"
format_stamp: "Formatted at 2026-08-10 14:41:25 on dist-test-slave-rmhg"
server_key: "6da9fcfb1b980367660f2eaacd70bf4f"
server_key_iv: "a53ae398aa238ee73d554e6e4f2d9f4c"
server_key_version: "encryptionkey@0"
I20260810 14:41:25.061816 32117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee"
format_stamp: "Formatted at 2026-08-10 14:41:25 on dist-test-slave-rmhg"
server_key: "6da9fcfb1b980367660f2eaacd70bf4f"
server_key_iv: "a53ae398aa238ee73d554e6e4f2d9f4c"
server_key_version: "encryptionkey@0"
I20260810 14:41:25.065824 32117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:41:25.068387 32140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:25.069777 32117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:41:25.069901 32117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee"
format_stamp: "Formatted at 2026-08-10 14:41:25 on dist-test-slave-rmhg"
server_key: "6da9fcfb1b980367660f2eaacd70bf4f"
server_key_iv: "a53ae398aa238ee73d554e6e4f2d9f4c"
server_key_version: "encryptionkey@0"
I20260810 14:41:25.070010 32117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:41:25.088383 32117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:41:25.091512 32117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:41:25.091724 32117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:41:25.092376 32117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:41:25.093362 32117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:41:25.093430 32117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:25.093487 32117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:41:25.093511 32117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:41:25.102779 32117 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:44409
I20260810 14:41:25.102799 32320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:44409 every 8 connection(s)
I20260810 14:41:25.103752 32117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:41:25.110036 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 32117
I20260810 14:41:25.110150 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:41:25.110438 15696 external_mini_cluster.cc:1428] Setting key 4783d6d131b2294d4c250480e75a9565
Aug 10 14:41:25 dist-test-slave-rmhg krb5kdc[31539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372885, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:41:25.119526 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.105703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:50985 (local address 127.15.84.62:42375)
0810 14:41:25.105837 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:25.105841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:25.106779 (+ 938us) server_negotiation.cc:407] Connection header received
0810 14:41:25.108219 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:25.108223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:25.108284 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:25.108361 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:25.109718 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.110306 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.111679 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.111910 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.112001 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:25.114545 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:25.114549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:25.114551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:41:25.114578 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:25.116233 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:25.117384 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:25.117389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:25.117391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:25.117511 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:25.118031 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:25.118035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:25.118036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:25.118299 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:25.118399 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:25.119174 (+ 775us) server_negotiation.cc:299] Negotiation successful
0810 14:41:25.119336 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:41:25.120795 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.106128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:42375 (local address 127.15.84.3:50985)
0810 14:41:25.106612 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:25.106667 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:25.107981 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:25.108559 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:25.108567 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:25.108969 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:41:25.109560 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:25.109576 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.110445 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.110448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:25.111565 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:25.111576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.112062 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.112066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:25.112247 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:25.112822 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:41:25.112840 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:41:25.114363 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:41:25.116758 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:41:25.116768 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:41:25.116790 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:41:25.117261 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:41:25.117638 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:41:25.117642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:41:25.117644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:41:25.117912 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:41:25.118417 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:41:25.118426 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:41:25.118864 (+ 438us) client_negotiation.cc:769] Sending connection context
0810 14:41:25.119128 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 14:41:25.119561 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":120,"threads_started":1}
I20260810 14:41:25.122282 32321 heartbeater.cc:347] Connected to a master server at 127.15.84.62:42375
I20260810 14:41:25.122581 32321 heartbeater.cc:464] Registering TS with master...
I20260810 14:41:25.123195 32321 heartbeater.cc:511] Master 127.15.84.62:42375 requested a full tablet report, sending...
I20260810 14:41:25.125438 31607 ts_manager.cc:194] Registered new tserver with Master: 3c82a0ff2f9d4b79a94df84b65ee9cee (127.15.84.3:44409)
I20260810 14:41:25.126191 31607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:50985
I20260810 14:41:25.126751 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:41:25.136665 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.129117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.15.84.62:42375)
0810 14:41:25.129391 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:25.129397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:25.129575 (+ 178us) server_negotiation.cc:407] Connection header received
0810 14:41:25.129710 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:25.129713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:25.129761 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:25.129826 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:41:25.130554 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.131080 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.131937 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.132170 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.132259 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:25.133700 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:41:25.133705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:41:25.133707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:41:25.133735 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:41:25.134923 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:25.135401 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:25.135405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:25.135407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:25.135490 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:41:25.135744 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:41:25.135748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:41:25.135750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:41:25.135994 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:41:25.136122 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:25.136370 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 14:41:25.136493 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 14:41:25.139793 31607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:41:25.143042 31607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:41:25.159164 32340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.153143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:46361 (local address 127.0.0.1:54722)
0810 14:41:25.153756 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:25.153770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:25.153859 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:25.154402 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:25.154405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:25.154424 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:41:25.154528 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:25.154534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.155867 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.155870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:25.157539 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:25.157548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.158539 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.158543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:25.158626 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:25.158644 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 14:41:25.158769 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 14:41:25.158963 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":243,"thread_start_us":119,"threads_started":1}
I20260810 14:41:25.159197 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.152448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.15.84.1:41745)
0810 14:41:25.152837 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:25.152856 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:25.153080 (+ 224us) server_negotiation.cc:407] Connection header received
0810 14:41:25.153401 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:25.153408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:25.153650 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:25.153818 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:41:25.154397 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.155665 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.157717 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.158612 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.158716 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:25.158886 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:25.158961 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 14:41:25.159062 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":161,"threads_started":1}
I20260810 14:41:25.159539 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.153447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.15.84.2:46361)
0810 14:41:25.153938 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:25.153955 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:25.153980 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:41:25.154055 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:25.154060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:25.154271 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:25.154430 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:41:25.154622 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.155732 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.157689 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.158405 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.159057 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:25.159235 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:25.159288 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 14:41:25.159387 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":121,"threads_started":1}
I20260810 14:41:25.159781 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.153949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.15.84.3:44409)
0810 14:41:25.154292 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:25.154303 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:25.154545 (+ 242us) server_negotiation.cc:407] Connection header received
0810 14:41:25.154630 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:25.154635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:25.154794 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:25.154924 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:41:25.155148 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.155990 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.157332 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.158247 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.158421 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:25.158595 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:25.159516 (+ 921us) server_negotiation.cc:299] Negotiation successful
0810 14:41:25.159623 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":141,"threads_started":1}
I20260810 14:41:25.160040 32338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.153516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:44409 (local address 127.0.0.1:32854)
0810 14:41:25.154358 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:25.154371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:25.154454 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:25.154916 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:25.154919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:25.154936 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:41:25.155033 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:25.155039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.156129 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.156132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:25.157088 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:25.157101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.158424 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.158427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:25.158567 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:25.158619 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 14:41:25.158785 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 14:41:25.158963 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":5,"thread_start_us":98,"threads_started":1}
I20260810 14:41:25.160876 32335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.152332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:41745 (local address 127.0.0.1:35038)
0810 14:41:25.152918 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:25.152984 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:25.153129 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:25.154045 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:25.154052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:25.154079 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:41:25.154252 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:25.154259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.155832 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.155835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:25.157568 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:25.157580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.158729 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.158733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:25.158833 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:25.158857 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:41:25.158917 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 14:41:25.158998 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":192,"thread_start_us":94,"threads_started":1}
I20260810 14:41:25.162776 32000 tablet_service.cc:1467] Processing CreateTablet for tablet df051f2362f14df983dc95ba2edeb938 (DEFAULT_TABLE table=test-table [id=ecf6506210a9481d8d98216de4ad9972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:41:25.162889 32217 tablet_service.cc:1467] Processing CreateTablet for tablet df051f2362f14df983dc95ba2edeb938 (DEFAULT_TABLE table=test-table [id=ecf6506210a9481d8d98216de4ad9972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:41:25.163889 32000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df051f2362f14df983dc95ba2edeb938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:25.163998 32217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df051f2362f14df983dc95ba2edeb938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:25.163627 31801 tablet_service.cc:1467] Processing CreateTablet for tablet df051f2362f14df983dc95ba2edeb938 (DEFAULT_TABLE table=test-table [id=ecf6506210a9481d8d98216de4ad9972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:41:25.164618 31801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df051f2362f14df983dc95ba2edeb938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:41:25.171518 32344 tablet_bootstrap.cc:492] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8: Bootstrap starting.
I20260810 14:41:25.171700 32345 tablet_bootstrap.cc:492] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee: Bootstrap starting.
I20260810 14:41:25.172559 32347 tablet_bootstrap.cc:492] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a: Bootstrap starting.
I20260810 14:41:25.174177 32345 tablet_bootstrap.cc:654] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:25.174623 32347 tablet_bootstrap.cc:654] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:25.174620 32344 tablet_bootstrap.cc:654] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8: Neither blocks nor log segments found. Creating new log.
I20260810 14:41:25.175431 32347 log.cc:824] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:41:25.175511 32344 log.cc:824] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8: Log is configured to *not* fsync() on all Append() calls
I20260810 14:41:25.175511 32345 log.cc:824] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee: Log is configured to *not* fsync() on all Append() calls
I20260810 14:41:25.177568 32347 tablet_bootstrap.cc:492] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a: No bootstrap required, opened a new log
I20260810 14:41:25.177780 32347 ts_tablet_manager.cc:1397] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 14:41:25.177825 32345 tablet_bootstrap.cc:492] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee: No bootstrap required, opened a new log
I20260810 14:41:25.177820 32344 tablet_bootstrap.cc:492] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8: No bootstrap required, opened a new log
I20260810 14:41:25.178015 32344 ts_tablet_manager.cc:1397] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 14:41:25.178014 32345 ts_tablet_manager.cc:1397] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:41:25.181326 32347 raft_consensus.cc:348] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf166d73d05a455f8a7072f198d00a2a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41745 } } peers { permanent_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46361 } } peers { permanent_uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 44409 } }
I20260810 14:41:25.181668 32347 raft_consensus.cc:374] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:25.181736 32347 raft_consensus.cc:729] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf166d73d05a455f8a7072f198d00a2a, State: Initialized, Role: FOLLOWER
I20260810 14:41:25.182003 32345 raft_consensus.cc:348] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf166d73d05a455f8a7072f198d00a2a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41745 } } peers { permanent_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46361 } } peers { permanent_uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 44409 } }
I20260810 14:41:25.182003 32344 raft_consensus.cc:348] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf166d73d05a455f8a7072f198d00a2a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41745 } } peers { permanent_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46361 } } peers { permanent_uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 44409 } }
I20260810 14:41:25.182279 32344 raft_consensus.cc:374] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:25.182279 32345 raft_consensus.cc:374] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:41:25.182331 32345 raft_consensus.cc:729] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c82a0ff2f9d4b79a94df84b65ee9cee, State: Initialized, Role: FOLLOWER
I20260810 14:41:25.182331 32344 raft_consensus.cc:729] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e1a418ce3a4394a3f1b3e081eca2a8, State: Initialized, Role: FOLLOWER
I20260810 14:41:25.182322 32347 consensus_queue.cc:260] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf166d73d05a455f8a7072f198d00a2a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41745 } } peers { permanent_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46361 } } peers { permanent_uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 44409 } }
I20260810 14:41:25.182801 32345 consensus_queue.cc:260] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf166d73d05a455f8a7072f198d00a2a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41745 } } peers { permanent_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46361 } } peers { permanent_uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 44409 } }
I20260810 14:41:25.182799 32344 consensus_queue.cc:260] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf166d73d05a455f8a7072f198d00a2a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41745 } } peers { permanent_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46361 } } peers { permanent_uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 44409 } }
I20260810 14:41:25.183539 32347 ts_tablet_manager.cc:1428] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:41:25.183854 32344 ts_tablet_manager.cc:1428] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260810 14:41:25.183854 32345 ts_tablet_manager.cc:1428] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:41:25.183959 31896 heartbeater.cc:503] Master 127.15.84.62:42375 was elected leader, sending a full tablet report...
I20260810 14:41:25.184218 32111 heartbeater.cc:503] Master 127.15.84.62:42375 was elected leader, sending a full tablet report...
I20260810 14:41:25.184666 32321 heartbeater.cc:503] Master 127.15.84.62:42375 was elected leader, sending a full tablet report...
I20260810 14:41:25.302351 32354 raft_consensus.cc:482] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:41:25.302588 32354 raft_consensus.cc:504] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf166d73d05a455f8a7072f198d00a2a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41745 } } peers { permanent_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46361 } } peers { permanent_uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 44409 } }
I20260810 14:41:25.303960 32354 leader_election.cc:288] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf166d73d05a455f8a7072f198d00a2a (127.15.84.1:41745), 3c82a0ff2f9d4b79a94df84b65ee9cee (127.15.84.3:44409)
I20260810 14:41:25.309098 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.304734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:40103 (local address 127.15.84.3:44409)
0810 14:41:25.304868 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:25.304873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:25.305080 (+ 207us) server_negotiation.cc:407] Connection header received
0810 14:41:25.305314 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:25.305318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:25.305371 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:25.305443 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:41:25.305685 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.306398 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.307812 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.308441 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.308553 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:25.308625 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:25.308879 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 14:41:25.308953 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:41:25.309098 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.304393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:48193 (local address 127.15.84.1:41745)
0810 14:41:25.304525 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:25.304529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:25.304546 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:41:25.304867 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:25.304871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:25.304954 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:25.305040 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:41:25.305408 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.306175 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.307825 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.308446 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.308553 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:25.308624 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:25.308879 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 14:41:25.308953 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 14:41:25.309121 32114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.304240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:41745 (local address 127.15.84.2:48193)
0810 14:41:25.304404 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:25.304422 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:25.304557 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:25.305071 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:25.305075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:25.305101 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:41:25.305292 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:25.305300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.306296 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.306299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:25.307686 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:25.307700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.308567 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.308571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:25.308743 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:25.308778 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 14:41:25.308855 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:41:25.308954 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":119,"spinlock_wait_cycles":80128}
I20260810 14:41:25.309129 32360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.304594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:44409 (local address 127.15.84.2:40103)
0810 14:41:25.305000 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:25.305014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:41:25.305195 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:41:25.305449 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:41:25.305451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:41:25.305466 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:41:25.305569 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:25.305576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.306493 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.306496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:25.307690 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:41:25.307700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.308567 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.308571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:41:25.308732 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:25.308787 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 14:41:25.308854 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 14:41:25.308953 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":73,"thread_start_us":147,"threads_started":1}
I20260810 14:41:25.309947 31828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df051f2362f14df983dc95ba2edeb938" candidate_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf166d73d05a455f8a7072f198d00a2a" is_pre_election: true
I20260810 14:41:25.309943 32247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df051f2362f14df983dc95ba2edeb938" candidate_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee" is_pre_election: true
I20260810 14:41:25.310316 32247 raft_consensus.cc:2454] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08e1a418ce3a4394a3f1b3e081eca2a8 in term 0.
I20260810 14:41:25.310316 31828 raft_consensus.cc:2454] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08e1a418ce3a4394a3f1b3e081eca2a8 in term 0.
I20260810 14:41:25.310825 31933 leader_election.cc:302] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08e1a418ce3a4394a3f1b3e081eca2a8, 3c82a0ff2f9d4b79a94df84b65ee9cee; no voters:
I20260810 14:41:25.311101 32354 raft_consensus.cc:2781] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:41:25.311198 32354 raft_consensus.cc:482] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:41:25.311251 32354 raft_consensus.cc:3037] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:25.312330 32354 raft_consensus.cc:504] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf166d73d05a455f8a7072f198d00a2a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41745 } } peers { permanent_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46361 } } peers { permanent_uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 44409 } }
I20260810 14:41:25.312690 32354 leader_election.cc:288] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [CANDIDATE]: Term 1 election: Requested vote from peers cf166d73d05a455f8a7072f198d00a2a (127.15.84.1:41745), 3c82a0ff2f9d4b79a94df84b65ee9cee (127.15.84.3:44409)
I20260810 14:41:25.313187 31828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df051f2362f14df983dc95ba2edeb938" candidate_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf166d73d05a455f8a7072f198d00a2a"
I20260810 14:41:25.313187 32247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df051f2362f14df983dc95ba2edeb938" candidate_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee"
I20260810 14:41:25.313331 31828 raft_consensus.cc:3037] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:25.313331 32247 raft_consensus.cc:3037] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:41:25.314695 31828 raft_consensus.cc:2454] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08e1a418ce3a4394a3f1b3e081eca2a8 in term 1.
I20260810 14:41:25.314697 32247 raft_consensus.cc:2454] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08e1a418ce3a4394a3f1b3e081eca2a8 in term 1.
I20260810 14:41:25.315023 31933 leader_election.cc:302] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08e1a418ce3a4394a3f1b3e081eca2a8, cf166d73d05a455f8a7072f198d00a2a; no voters:
I20260810 14:41:25.315234 32354 raft_consensus.cc:2781] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:41:25.315502 32354 raft_consensus.cc:686] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [term 1 LEADER]: Becoming Leader. State: Replica: 08e1a418ce3a4394a3f1b3e081eca2a8, State: Running, Role: LEADER
I20260810 14:41:25.315861 32354 consensus_queue.cc:237] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf166d73d05a455f8a7072f198d00a2a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41745 } } peers { permanent_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46361 } } peers { permanent_uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 44409 } }
I20260810 14:41:25.319314 31600 catalog_manager.cc:5495] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08e1a418ce3a4394a3f1b3e081eca2a8 (127.15.84.2). New cstate: current_term: 1 leader_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf166d73d05a455f8a7072f198d00a2a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08e1a418ce3a4394a3f1b3e081eca2a8" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 46361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 44409 } health_report { overall_health: UNKNOWN } } }
I20260810 14:41:25.351022 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.347678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54726 (local address 127.15.84.2:46361)
0810 14:41:25.347802 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 14:41:25.347808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:41:25.347825 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:41:25.347965 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:41:25.347967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:41:25.348024 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:41:25.348119 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:41:25.348339 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.348850 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.349727 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:41:25.349946 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:41:25.350023 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:41:25.350276 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:41:25.350728 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:41:25.350819 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 14:41:25.350848 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 14:41:25.350905 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 14:41:25.354835 32322 tablet.cc:2369] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:41:25.357651 31828 raft_consensus.cc:1261] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a [term 1 FOLLOWER]: Refusing update from remote peer 08e1a418ce3a4394a3f1b3e081eca2a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:41:25.357649 32247 raft_consensus.cc:1261] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee [term 1 FOLLOWER]: Refusing update from remote peer 08e1a418ce3a4394a3f1b3e081eca2a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:41:25.358474 32354 consensus_queue.cc:1035] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf166d73d05a455f8a7072f198d00a2a" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:41:25.358722 32362 consensus_queue.cc:1035] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c82a0ff2f9d4b79a94df84b65ee9cee" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 44409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:41:25.375617 31603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:41:25.375830 31603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:41:25.378347 31603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 417370ca1a81496683dda9a3a1ce6902: Sending DeleteTablet for 3 replicas of tablet df051f2362f14df983dc95ba2edeb938
I20260810 14:41:25.379138 31801 tablet_service.cc:1514] Processing DeleteTablet for tablet df051f2362f14df983dc95ba2edeb938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:41:25 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:35038
I20260810 14:41:25.379256 32217 tablet_service.cc:1514] Processing DeleteTablet for tablet df051f2362f14df983dc95ba2edeb938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:41:25 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:32854
I20260810 14:41:25.379701 32380 tablet_replica.cc:330] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a: stopping tablet replica
I20260810 14:41:25.379947 32382 tablet_replica.cc:330] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee: stopping tablet replica
I20260810 14:41:25.380041 32000 tablet_service.cc:1514] Processing DeleteTablet for tablet df051f2362f14df983dc95ba2edeb938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:41:25 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:54722
I20260810 14:41:25.380220 32382 raft_consensus.cc:2229] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:41:25.380236 32380 raft_consensus.cc:2229] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:41:25.380568 32382 raft_consensus.cc:2258] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:41:25.380568 32380 raft_consensus.cc:2258] T df051f2362f14df983dc95ba2edeb938 P cf166d73d05a455f8a7072f198d00a2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:41:25.380657 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 31697
I20260810 14:41:25.382213 32382 ts_tablet_manager.cc:1905] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:41:25.384552 32382 ts_tablet_manager.cc:1918] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:41:25.384663 32382 log.cc:1196] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/df051f2362f14df983dc95ba2edeb938
I20260810 14:41:25.385027 32382 ts_tablet_manager.cc:1939] T df051f2362f14df983dc95ba2edeb938 P 3c82a0ff2f9d4b79a94df84b65ee9cee: Deleting consensus metadata
I20260810 14:41:25.385046 32384 tablet_replica.cc:330] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8: stopping tablet replica
I20260810 14:41:25.385577 32384 raft_consensus.cc:2229] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:41:25.386052 32384 raft_consensus.cc:2258] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:41:25.386528 31586 catalog_manager.cc:4841] TS 3c82a0ff2f9d4b79a94df84b65ee9cee (127.15.84.3:44409): tablet df051f2362f14df983dc95ba2edeb938 (table test-table [id=ecf6506210a9481d8d98216de4ad9972]) successfully deleted
I20260810 14:41:25.394575 32384 ts_tablet_manager.cc:1905] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:41:25.397431 32384 ts_tablet_manager.cc:1918] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:41:25.397563 32384 log.cc:1196] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/df051f2362f14df983dc95ba2edeb938
I20260810 14:41:25.397874 32384 ts_tablet_manager.cc:1939] T df051f2362f14df983dc95ba2edeb938 P 08e1a418ce3a4394a3f1b3e081eca2a8: Deleting consensus metadata
I20260810 14:41:25.399180 31586 catalog_manager.cc:4841] TS 08e1a418ce3a4394a3f1b3e081eca2a8 (127.15.84.2:46361): tablet df051f2362f14df983dc95ba2edeb938 (table test-table [id=ecf6506210a9481d8d98216de4ad9972]) successfully deleted
W20260810 14:41:25.405220 31586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:41745 (error 108)
I20260810 14:41:25.405465 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 31905
I20260810 14:41:25.420459 32335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:41:25.420039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:41745 (local address 127.0.0.1:35044)
0810 14:41:25.420258 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 14:41:25.420346 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260810 14:41:25.420938 31586 catalog_manager.cc:4584] TS cf166d73d05a455f8a7072f198d00a2a (127.15.84.1:41745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df051f2362f14df983dc95ba2edeb938: Network error: Client connection negotiation failed: client connection to 127.15.84.1:41745: connect: Connection refused (error 111)
I20260810 14:41:25.429105 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 32117
I20260810 14:41:25.439301 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 31557
2026-08-10T14:41:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2583 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:41:25 dist-test-slave-rmhg krb5kdc[32390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:41:25 dist-test-slave-rmhg krb5kdc[32390](info): set up 2 sockets
Aug 10 14:41:25 dist-test-slave-rmhg krb5kdc[32390](info): commencing operation
krb5kdc: starting...
W20260810 14:41:26.616704 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:41:26 dist-test-slave-rmhg krb5kdc[32390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:41:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:41:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/rangeradmin_127.15.84.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/rangeradmin_127.15.84.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/rangerlookup_127.15.84.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/rangerlookup_127.15.84.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/HTTP_127.15.84.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/HTTP_127.15.84.20@KRBTEST.COM.keytab.
I20260810 14:41:26.804670 15696 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPiaUUw/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/postgres -l logfile start
2026-08-10 14:41:29.231 UTC [32432] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 14:41:29.232 UTC [32432] LOG: listening on IPv4 address "127.15.84.20", port 54495
2026-08-10 14:41:29.235 UTC [32432] LOG: listening on Unix socket "/tmp/.s.PGSQL.54495"
2026-08-10 14:41:29.256 UTC [32436] LOG: database system was shut down at 2026-08-10 14:41:27 UTC
2026-08-10 14:41:29.259 UTC [32432] LOG: database system is ready to accept connections
I20260810 14:41:30.312088 15696 mini_postgres.cc:96] Postgres bound to 54495
2026-08-10 14:41:30.320 UTC [32444] FATAL: database "slave" does not exist
127.15.84.20:54495 - accepting connections
I20260810 14:41:30.321286 15696 mini_ranger.cc:161] Starting Ranger...
I20260810 14:41:30.339170 15696 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 14:41:30.788800 15696 mini_ranger.cc:87] Created ranger Postgres database
I20260810 14:41:30.788890 15696 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-admin
2026-08-10 14:41:31,147 [I] DB FLAVOR :POSTGRES
2026-08-10 14:41:31,147 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:41:31,147 [I] Checking connection..
2026-08-10 14:41:31,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 14:41:31,405 [I] Checking connection passed.
2026-08-10 14:41:31,405 [I] --------- Verifying version history table ---------
2026-08-10 14:41:31,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:41:31,656 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:41:31,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:41:31,922 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:41:31,922 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 14:41:31,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 14:41:32.188 UTC [32546] WARNING: there is no transaction in progress
2026-08-10 14:41:32,195 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 14:41:32,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:41:32,476 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 14:41:32,476 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:41:32,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 14:41:32,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 14:41:33,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rmhg.c.gcp-upstream.internal','N') ;"
2026-08-10 14:41:33,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:41:33,539 [I] Table x_portal_user does not exist in database ranger
2026-08-10 14:41:33,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:41:33,809 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 14:41:33,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:41:34,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:41:34,304 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 14:41:34,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 14:41:35.268 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.284 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.301 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.317 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.334 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.351 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.407 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.414 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.428 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.438 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.448 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.462 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.472 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.479 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.485 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:35.491 UTC [32719] WARNING: there is no transaction in progress
2026-08-10 14:41:36,128 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 14:41:36,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 14:41:36,395 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 14:41:36,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:41:36,640 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 14:41:36,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:41:36,915 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 14:41:36,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:41:37,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:41:37,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rmhg.c.gcp-upstream.internal';"
2026-08-10 14:41:37,701 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 14:41:37,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 14:41:37,974 [I] DB_PATCHES have already been applied
I20260810 14:41:37.980427 15696 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPiaUUw/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 14:41:37.980499 15696 mini_ranger.cc:193] Using host: 127.15.84.20
I20260810 14:41:37.984905 15696 mini_ranger.cc:227] Ranger admin URL: http://127.15.84.20:54775
Aug 10, 2026 2:41:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:41:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:41:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 2:41:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 2:41:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 2:41:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 2:41:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 14:41:39.244650 15696 mini_ranger.cc:160] Time spent starting Ranger: real 8.923s user 0.002s sys 0.010s
Aug 10, 2026 2:41:58 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 2:41:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 2:41:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 2:41:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 2:41:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 2:41:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 2:41:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 2:41:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 2:41:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 2:41:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 2:41:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 2:41:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 2:41:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 2:41:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 2:41:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 2:41:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 2:41:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 2:42:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 14:42:00.677569 15696 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/rangerkms_127.15.84.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/rangerkms_127.15.84.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.84.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.84.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.84.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/HTTP_127.15.84.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.84.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/HTTP_127.15.84.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 14:42:00 dist-test-slave-rmhg krb5kdc[32390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372920, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 14:42:00.773090 15696 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 14:42:00.784283 15696 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 14:42:01.541406 15696 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 14:42:01.541500 15696 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-kms
I20260810 14:42:01.541594 15696 mini_ranger_kms.cc:227] Using postgres at 127.15.84.20:54495
2026-08-10 14:42:01,812 [I] DB FLAVOR :POSTGRES
2026-08-10 14:42:01,812 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:42:01,812 [I] Checking connection
2026-08-10 14:42:01,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 14:42:02,059 [I] connection success
2026-08-10 14:42:02,059 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 14:42:02,059 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 14:42:02,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 14:42:02,307 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 14:42:02,307 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:42:02,307 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 14:42:02,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:54495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 14:42:02,582 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 14:42:03.084307 15696 mini_ranger_kms.cc:310] Created kms service
I20260810 14:42:03.292317 15696 mini_ranger_kms.cc:326] Created kudu user
I20260810 14:42:03.342284 15696 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 14:42:03 dist-test-slave-rmhg krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372920, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:42:03.781098 15696 mini_ranger_kms.cc:384] Added ranger policy
I20260810 14:42:03.781304 15696 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 14:42:03.781334 15696 mini_ranger_kms.cc:240] Using host: 127.15.84.20
I20260810 14:42:03.786516 15696 mini_ranger_kms.cc:276] Ranger KMS PID: 544
I20260810 14:42:03.786698 15696 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.84.20:59813
Aug 10, 2026 2:42:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:42:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:42:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 2:42:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 2:42:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 2:42:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 14:42:05 dist-test-slave-rmhg krb5kdc[32390](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372925, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.84.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:42:06.319689 15696 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.547s user 0.036s sys 0.193s
I20260810 14:42:06.319833 15696 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 14:42:06.319962 15696 mini_ranger_kms.cc:401] 127.15.84.20:59813/kms/v1/keys
Aug 10 14:42:06 dist-test-slave-rmhg krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372925, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.84.20@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
Aug 10 14:42:06 dist-test-slave-rmhg krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372925, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.84.20@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
Aug 10 14:42:09 dist-test-slave-rmhg krb5kdc[32390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:42:09.166139 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:41607
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:41607
--ranger_config_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.84.20:59813/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:42:09.283779 625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:42:09.284094 625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:42:09.284152 625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:42:09.288355 625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:42:09.288445 625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:42:09.288475 625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:42:09.288496 625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:42:09.288514 625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:42:09.294327 625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.84.20:59813/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:41607
--ranger_config_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:41607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:42:09.295517 625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:42:09.296615 625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:42:09.305605 634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:09.305610 637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:09.305610 639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:42:09.305912 625 server_base.cc:1034] running on GCE node
I20260810 14:42:09.306595 625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:42:09.307803 625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:42:09.308980 625 hybrid_clock.cc:648] HybridClock initialized: now 1786372929308966 us; error 29 us; skew 500 ppm
Aug 10 14:42:09 dist-test-slave-rmhg krb5kdc[32390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372929, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:42:09.311964 625 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:42:09.313014 625 webserver.cc:460] Webserver started at http://127.15.84.62:32829/ using document root <none> and password file <none>
I20260810 14:42:09.313625 625 fs_manager.cc:359] Metadata directory not provided
I20260810 14:42:09.313690 625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:42:09.313905 625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:42:09 dist-test-slave-rmhg krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372929, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:42:09.372064 625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "547dfc791cb248cfb318c1c3df2ca087"
format_stamp: "Formatted at 2026-08-10 14:42:09 on dist-test-slave-rmhg"
server_key: "8004691b41c74b8310e15853a5b36c4a"
server_key_iv: "d169e8f90c0d58e11e0987593ccdaf73"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:09.372852 625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "547dfc791cb248cfb318c1c3df2ca087"
format_stamp: "Formatted at 2026-08-10 14:42:09 on dist-test-slave-rmhg"
server_key: "8004691b41c74b8310e15853a5b36c4a"
server_key_iv: "d169e8f90c0d58e11e0987593ccdaf73"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:09.377260 625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:42:09.395910 649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:09.397874 625 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 14:42:09.398054 625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "547dfc791cb248cfb318c1c3df2ca087"
format_stamp: "Formatted at 2026-08-10 14:42:09 on dist-test-slave-rmhg"
server_key: "8004691b41c74b8310e15853a5b36c4a"
server_key_iv: "d169e8f90c0d58e11e0987593ccdaf73"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:09.398227 625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:42:09.432206 625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:42:09.435431 625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:42:09.435616 625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:42:09.443023 625 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:41607
I20260810 14:42:09.443050 745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:41607 every 8 connection(s)
I20260810 14:42:09.444166 625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:42:09.445727 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 625
I20260810 14:42:09.445844 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:42:09.446122 15696 external_mini_cluster.cc:1428] Setting key aa2e43316bed61a93acb72798f994660
I20260810 14:42:09.447561 746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:42:09.454622 746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087: Bootstrap starting.
Aug 10 14:42:09 dist-test-slave-rmhg krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:42:09.457461 746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087: Neither blocks nor log segments found. Creating new log.
I20260810 14:42:09.458441 746 log.cc:824] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087: Log is configured to *not* fsync() on all Append() calls
I20260810 14:42:09.460997 746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087: No bootstrap required, opened a new log
I20260810 14:42:09.462682 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:09.447609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.15.84.62:41607)
0810 14:42:09.448082 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:09.448102 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:09.448146 (+ 44us) server_negotiation.cc:407] Connection header received
0810 14:42:09.448996 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:09.449020 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:09.449481 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:09.449877 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:42:09.450536 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:09.452027 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:09.453061 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:09.453803 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:09.453957 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:09.456800 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:42:09.456807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:42:09.456824 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:42:09.456860 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:42:09.459483 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:09.459957 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:09.459963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:09.459969 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:09.460124 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:09.460354 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:09.460357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:09.460359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:09.460914 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:42:09.461126 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:09.461400 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 14:42:09.461714 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":127,"threads_started":1}
I20260810 14:42:09.464906 746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547dfc791cb248cfb318c1c3df2ca087" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 41607 } }
I20260810 14:42:09.465240 746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:42:09.465291 746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547dfc791cb248cfb318c1c3df2ca087, State: Initialized, Role: FOLLOWER
I20260810 14:42:09.465708 746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547dfc791cb248cfb318c1c3df2ca087" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 41607 } }
I20260810 14:42:09.465832 746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:42:09.465879 746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:42:09.465951 746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:42:09.466845 746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547dfc791cb248cfb318c1c3df2ca087" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 41607 } }
I20260810 14:42:09.467164 746 leader_election.cc:302] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 547dfc791cb248cfb318c1c3df2ca087; no voters:
I20260810 14:42:09.467437 746 leader_election.cc:288] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:42:09.467581 757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:42:09.467832 757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [term 1 LEADER]: Becoming Leader. State: Replica: 547dfc791cb248cfb318c1c3df2ca087, State: Running, Role: LEADER
I20260810 14:42:09.468132 757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547dfc791cb248cfb318c1c3df2ca087" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 41607 } }
I20260810 14:42:09.468680 746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:42:09.469791 758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "547dfc791cb248cfb318c1c3df2ca087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547dfc791cb248cfb318c1c3df2ca087" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 41607 } } }
I20260810 14:42:09.469923 758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [sys.catalog]: This master's current role is: LEADER
I20260810 14:42:09.470486 767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:42:09.470570 759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 547dfc791cb248cfb318c1c3df2ca087. Latest consensus state: current_term: 1 leader_uuid: "547dfc791cb248cfb318c1c3df2ca087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547dfc791cb248cfb318c1c3df2ca087" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 41607 } } }
I20260810 14:42:09.470757 759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087 [sys.catalog]: This master's current role is: LEADER
I20260810 14:42:09.471042 746 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 14:42:09.477216 767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:42:09.484875 767 catalog_manager.cc:1347] Generated new cluster ID: be75eace8f224dd69e29d0cb0bc80b51
I20260810 14:42:09.484953 767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:42:09.509702 767 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:42:09.511214 767 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:42:09.519953 767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087: Generated new TSK 0
I20260810 14:42:09.520912 767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:42:10.226169 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.84.20:59813/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:41607
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:42:10.374305 814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:42:10.374720 814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:42:10.374784 814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:42:10.381487 814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:42:10.381585 814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:42:10.381675 814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:42:10.388707 814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.84.20:59813/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:41607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:42:10.389984 814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:42:10.391194 814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:42:10.401299 827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:10.401434 835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:10.401314 826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:42:10.402453 814 server_base.cc:1034] running on GCE node
I20260810 14:42:10.402958 814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:42:10.403703 814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:42:10.404881 814 hybrid_clock.cc:648] HybridClock initialized: now 1786372930404851 us; error 50 us; skew 500 ppm
Aug 10 14:42:10 dist-test-slave-rmhg krb5kdc[32390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372930, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:42:10.408012 814 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:42:10.409094 814 webserver.cc:460] Webserver started at http://127.15.84.1:44581/ using document root <none> and password file <none>
I20260810 14:42:10.409739 814 fs_manager.cc:359] Metadata directory not provided
I20260810 14:42:10.409816 814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:42:10.410036 814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:42:10 dist-test-slave-rmhg krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372930, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:42:10.427066 814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "6a64301b6bf04c7dbf02e60333f1d61e"
format_stamp: "Formatted at 2026-08-10 14:42:10 on dist-test-slave-rmhg"
server_key: "c19186f23f5bc4ac4362b45100d64ad5"
server_key_iv: "7271f5a1ba201b9ec90aa5dd44a012ad"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:10.427865 814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "6a64301b6bf04c7dbf02e60333f1d61e"
format_stamp: "Formatted at 2026-08-10 14:42:10 on dist-test-slave-rmhg"
server_key: "c19186f23f5bc4ac4362b45100d64ad5"
server_key_iv: "7271f5a1ba201b9ec90aa5dd44a012ad"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:10.439118 814 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.003s sys 0.004s
I20260810 14:42:10.466454 854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:10.468673 814 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:42:10.468839 814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "6a64301b6bf04c7dbf02e60333f1d61e"
format_stamp: "Formatted at 2026-08-10 14:42:10 on dist-test-slave-rmhg"
server_key: "c19186f23f5bc4ac4362b45100d64ad5"
server_key_iv: "7271f5a1ba201b9ec90aa5dd44a012ad"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:10.469013 814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:42:10.500640 814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:42:10.504837 814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:42:10.505127 814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:42:10.506148 814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:42:10.507433 814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:42:10.507568 814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 14:42:10.507696 814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:42:10.507777 814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:10.519199 814 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:42897
I20260810 14:42:10.519256 1030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:42897 every 8 connection(s)
I20260810 14:42:10.521111 814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:42:10.531307 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 814
I20260810 14:42:10.531461 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:42:10.531780 15696 external_mini_cluster.cc:1428] Setting key ebbbacd81571ee8669489e7b2afc60ff
Aug 10 14:42:10 dist-test-slave-rmhg krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372930, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:42:10.540148 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:10.523881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:41607 (local address 127.15.84.1:47635)
0810 14:42:10.524398 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 14:42:10.524453 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:42:10.525556 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:42:10.526234 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:42:10.526243 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:42:10.526648 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:42:10.527344 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:10.527363 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:10.528569 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:10.528573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:10.530063 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:10.530076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:10.530774 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:10.530779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:10.531007 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:10.531807 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:42:10.531832 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:42:10.533838 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:42:10.536247 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:42:10.536256 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:42:10.536278 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:42:10.536718 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:42:10.537062 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:42:10.537065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:42:10.537067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:42:10.537312 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:42:10.537907 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:42:10.537915 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:42:10.538305 (+ 390us) client_negotiation.cc:769] Sending connection context
0810 14:42:10.538543 (+ 238us) client_negotiation.cc:240] Negotiation successful
0810 14:42:10.538930 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":111,"threads_started":1}
I20260810 14:42:10.541095 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:10.523491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:47635 (local address 127.15.84.62:41607)
0810 14:42:10.523800 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:10.523805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:10.524593 (+ 788us) server_negotiation.cc:407] Connection header received
0810 14:42:10.525847 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:10.525852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:10.525913 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:10.526017 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:42:10.527547 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:10.528412 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:10.530211 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:10.530579 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:10.530705 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:10.534073 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:42:10.534078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:42:10.534081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:42:10.534113 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:42:10.536088 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:10.536848 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:10.536852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:10.536854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:10.536954 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:10.537427 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:10.537430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:10.537432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:10.537746 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:42:10.537846 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:10.540748 (+ 2902us) server_negotiation.cc:299] Negotiation successful
0810 14:42:10.540905 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
I20260810 14:42:10.543183 1031 heartbeater.cc:347] Connected to a master server at 127.15.84.62:41607
I20260810 14:42:10.543525 1031 heartbeater.cc:464] Registering TS with master...
I20260810 14:42:10.544237 1031 heartbeater.cc:511] Master 127.15.84.62:41607 requested a full tablet report, sending...
I20260810 14:42:10.546265 683 ts_manager.cc:194] Registered new tserver with Master: 6a64301b6bf04c7dbf02e60333f1d61e (127.15.84.1:42897)
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:42:10.547914 683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:47635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
Aug 10 14:42:10 dist-test-slave-rmhg krb5kdc[32390](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372930, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:42:10.572018 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.84.20:59813/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:41607
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:42:10.739447 1040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:42:10.739858 1040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:42:10.739923 1040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:42:10.746111 1040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:42:10.746269 1040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:42:10.746397 1040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:42:10.754627 1040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.84.20:59813/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:41607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:42:10.756328 1040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:42:10.757825 1040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:42:10.771341 1054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:10.772137 1051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:10.772373 1059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:42:10.773305 1040 server_base.cc:1034] running on GCE node
I20260810 14:42:10.773916 1040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:42:10.774832 1040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:42:10.776139 1040 hybrid_clock.cc:648] HybridClock initialized: now 1786372930776112 us; error 51 us; skew 500 ppm
Aug 10 14:42:10 dist-test-slave-rmhg krb5kdc[32390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372930, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:42:10.780367 1040 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:42:10.781656 1040 webserver.cc:460] Webserver started at http://127.15.84.2:43963/ using document root <none> and password file <none>
I20260810 14:42:10.782431 1040 fs_manager.cc:359] Metadata directory not provided
I20260810 14:42:10.782528 1040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:42:10.782847 1040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:42:10 dist-test-slave-rmhg krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372930, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:42:10.799240 1040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "93f14190ada440d9a6fdcfcec564d221"
format_stamp: "Formatted at 2026-08-10 14:42:10 on dist-test-slave-rmhg"
server_key: "725d907dd7619b0847f56fda7c7d1aa3"
server_key_iv: "ea1b05835d6f1cdc8e0ced38a083419d"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:10.800017 1040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "93f14190ada440d9a6fdcfcec564d221"
format_stamp: "Formatted at 2026-08-10 14:42:10 on dist-test-slave-rmhg"
server_key: "725d907dd7619b0847f56fda7c7d1aa3"
server_key_iv: "ea1b05835d6f1cdc8e0ced38a083419d"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:10.805461 1040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260810 14:42:10.831796 1071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:10.833662 1040 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 14:42:10.833849 1040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "93f14190ada440d9a6fdcfcec564d221"
format_stamp: "Formatted at 2026-08-10 14:42:10 on dist-test-slave-rmhg"
server_key: "725d907dd7619b0847f56fda7c7d1aa3"
server_key_iv: "ea1b05835d6f1cdc8e0ced38a083419d"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:10.834038 1040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:42:10.860769 1040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:42:10.864779 1040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:42:10.864995 1040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:42:10.865773 1040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:42:10.866895 1040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:42:10.866962 1040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:10.867023 1040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:42:10.867054 1040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:10.878470 1040 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:35303
I20260810 14:42:10.878491 1260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:35303 every 8 connection(s)
I20260810 14:42:10.879596 1040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:42:10.888338 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 1040
I20260810 14:42:10.888497 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:42:10.888850 15696 external_mini_cluster.cc:1428] Setting key 5877ba57fd4bb1226ddf45f056573089
Aug 10 14:42:10 dist-test-slave-rmhg krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372930, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:42:10.896661 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:10.883391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:35219 (local address 127.15.84.62:41607)
0810 14:42:10.883521 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:10.883526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:10.884007 (+ 481us) server_negotiation.cc:407] Connection header received
0810 14:42:10.885195 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:10.885200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:10.885262 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:10.885323 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:42:10.886821 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:10.887366 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:10.888572 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:10.888820 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:10.888882 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:10.892065 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:42:10.892070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:42:10.892073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:42:10.892106 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:42:10.893927 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:10.894621 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:10.894625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:10.894627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:10.894737 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:10.895183 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:10.895187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:10.895188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:10.895435 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:42:10.895517 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:10.896333 (+ 816us) server_negotiation.cc:299] Negotiation successful
0810 14:42:10.896481 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:42:10.898690 1267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:10.882336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:41607 (local address 127.15.84.2:35219)
0810 14:42:10.883809 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0810 14:42:10.883888 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 14:42:10.884892 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:42:10.885789 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:42:10.885798 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:42:10.886130 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:42:10.886652 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:10.886666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:10.887500 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:10.887503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:10.888421 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:10.888434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:10.889080 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:10.889082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:10.889300 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:10.890010 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:42:10.890034 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:42:10.891823 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:42:10.894074 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:42:10.894081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:42:10.894102 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:42:10.894495 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:42:10.894858 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:42:10.894861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:42:10.894862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:42:10.895068 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:42:10.895610 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:42:10.895617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:42:10.896016 (+ 399us) client_negotiation.cc:769] Sending connection context
0810 14:42:10.897145 (+ 1129us) client_negotiation.cc:240] Negotiation successful
0810 14:42:10.897558 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":130,"threads_started":1}
I20260810 14:42:10.900476 1262 heartbeater.cc:347] Connected to a master server at 127.15.84.62:41607
I20260810 14:42:10.900774 1262 heartbeater.cc:464] Registering TS with master...
I20260810 14:42:10.901536 1262 heartbeater.cc:511] Master 127.15.84.62:41607 requested a full tablet report, sending...
I20260810 14:42:10.902832 683 ts_manager.cc:194] Registered new tserver with Master: 93f14190ada440d9a6fdcfcec564d221 (127.15.84.2:35303)
I20260810 14:42:10.903612 683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:35219
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:42:10.931981 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.84.20:59813/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:41607
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:42:10 dist-test-slave-rmhg krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372930, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
W20260810 14:42:11.102389 1276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:42:11.102823 1276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:42:11.102897 1276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:42:11.108719 1276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:42:11.108834 1276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:42:11.108923 1276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:42:11.117100 1276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.84.20:59813/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:41607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:42:11.118652 1276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:42:11.120188 1276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:42:11.134171 1286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:11.134225 1282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:11.135119 1281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:42:11.135619 1276 server_base.cc:1034] running on GCE node
I20260810 14:42:11.136152 1276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:42:11.137001 1276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:42:11.138187 1276 hybrid_clock.cc:648] HybridClock initialized: now 1786372931138153 us; error 50 us; skew 500 ppm
Aug 10 14:42:11 dist-test-slave-rmhg krb5kdc[32390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372931, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:42:11.141947 1276 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:42:11.143115 1276 webserver.cc:460] Webserver started at http://127.15.84.3:37599/ using document root <none> and password file <none>
I20260810 14:42:11.143818 1276 fs_manager.cc:359] Metadata directory not provided
I20260810 14:42:11.143899 1276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:42:11.144137 1276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:42:11 dist-test-slave-rmhg krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372931, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:42:11.172142 1276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "6b4a89dac3e14147bc2a0d46e297406a"
format_stamp: "Formatted at 2026-08-10 14:42:11 on dist-test-slave-rmhg"
server_key: "6235c015eb37b41c3c427bdf325195e6"
server_key_iv: "3107c59231a5f292b35096c9b687fd17"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:11.173049 1276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "6b4a89dac3e14147bc2a0d46e297406a"
format_stamp: "Formatted at 2026-08-10 14:42:11 on dist-test-slave-rmhg"
server_key: "6235c015eb37b41c3c427bdf325195e6"
server_key_iv: "3107c59231a5f292b35096c9b687fd17"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:11.181919 1276 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260810 14:42:11.202636 1301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:11.204599 1276 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:42:11.204756 1276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "6b4a89dac3e14147bc2a0d46e297406a"
format_stamp: "Formatted at 2026-08-10 14:42:11 on dist-test-slave-rmhg"
server_key: "6235c015eb37b41c3c427bdf325195e6"
server_key_iv: "3107c59231a5f292b35096c9b687fd17"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:11.204907 1276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:42:11.233640 1276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:42:11.237556 1276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:42:11.237797 1276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:42:11.238629 1276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:42:11.239867 1276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:42:11.239930 1276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:11.239984 1276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:42:11.240013 1276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:11.251945 1276 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:34513
I20260810 14:42:11.252002 1482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:34513 every 8 connection(s)
I20260810 14:42:11.253315 1276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:42:11.256402 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 1276
I20260810 14:42:11.256523 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:42:11.256801 15696 external_mini_cluster.cc:1428] Setting key 481fea3fc11d9e36166851f5187bbfcc
Aug 10 14:42:11 dist-test-slave-rmhg krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372931, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:42:11.272717 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.256034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:57249 (local address 127.15.84.62:41607)
0810 14:42:11.257237 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:11.257242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:11.258096 (+ 854us) server_negotiation.cc:407] Connection header received
0810 14:42:11.259388 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:11.259392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:11.259450 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:11.259504 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:42:11.261285 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.262080 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.263872 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.264223 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.264314 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:11.267561 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:42:11.267565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:42:11.267569 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:42:11.267602 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:42:11.269416 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:11.270071 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:11.270075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:11.270077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:11.270168 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:11.270580 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:11.270584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:11.270586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:11.270818 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:42:11.270892 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:11.272413 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0810 14:42:11.272558 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1043}
I20260810 14:42:11.274740 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.256581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:41607 (local address 127.15.84.3:57249)
0810 14:42:11.257924 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0810 14:42:11.257990 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 14:42:11.259114 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:42:11.259946 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:42:11.259956 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:42:11.260342 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:42:11.261056 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:11.261079 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.262267 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.262270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:11.263711 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:11.263725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.264573 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.264577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:11.264809 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:11.265578 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:42:11.265601 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:42:11.267345 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:42:11.269563 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:42:11.269571 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:42:11.269592 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:42:11.269951 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:42:11.270283 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:42:11.270286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:42:11.270289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:42:11.270478 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:42:11.270956 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:42:11.270964 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:42:11.271384 (+ 420us) client_negotiation.cc:769] Sending connection context
0810 14:42:11.273197 (+ 1813us) client_negotiation.cc:240] Negotiation successful
0810 14:42:11.273618 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"thread_start_us":159,"threads_started":1}
I20260810 14:42:11.276410 1483 heartbeater.cc:347] Connected to a master server at 127.15.84.62:41607
I20260810 14:42:11.276707 1483 heartbeater.cc:464] Registering TS with master...
I20260810 14:42:11.277506 1483 heartbeater.cc:511] Master 127.15.84.62:41607 requested a full tablet report, sending...
I20260810 14:42:11.278728 683 ts_manager.cc:194] Registered new tserver with Master: 6b4a89dac3e14147bc2a0d46e297406a (127.15.84.3:34513)
I20260810 14:42:11.279506 683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:57249
I20260810 14:42:11.286514 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:42:11.298832 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.289346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.15.84.62:41607)
0810 14:42:11.289474 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:11.289478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:11.289786 (+ 308us) server_negotiation.cc:407] Connection header received
0810 14:42:11.289939 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:11.289941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:11.289988 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:11.290049 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:42:11.290906 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.291442 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.292632 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.292867 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.292965 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:11.294806 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:42:11.294810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:42:11.294813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:42:11.294849 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:42:11.296695 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:11.297281 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:11.297285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:11.297287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:11.297390 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:11.297776 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:11.297780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:11.297782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:11.298043 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:42:11.298191 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:11.298508 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 14:42:11.298645 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:42:11.302589 683 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:42:11.307441 683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 10 14:42:11 dist-test-slave-rmhg krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372930, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:42:11.355818 1500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.339696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:35303 (local address 127.0.0.1:42366)
0810 14:42:11.346014 (+ 6318us) negotiation.cc:107] Waiting for socket to connect
0810 14:42:11.346029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:42:11.346117 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:42:11.349461 (+ 3344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:42:11.349465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:42:11.349490 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:42:11.349605 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:11.349611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.351073 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.351076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:11.352108 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:11.352122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.353968 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.353972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:11.355276 (+ 1304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:11.355343 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 14:42:11.355470 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 14:42:11.355647 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6257,"mutex_wait_us":32,"thread_start_us":108,"threads_started":1}
I20260810 14:42:11.355850 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.340972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.15.84.2:35303)
0810 14:42:11.344624 (+ 3652us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:11.344645 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:11.346224 (+ 1579us) server_negotiation.cc:407] Connection header received
0810 14:42:11.346312 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:11.346318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:11.346550 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:11.346714 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:42:11.349742 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.350927 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.352631 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.353600 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.353709 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:11.353887 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:11.355552 (+ 1665us) server_negotiation.cc:299] Negotiation successful
0810 14:42:11.355690 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3533,"thread_start_us":103,"threads_started":1}
I20260810 14:42:11.356480 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.339286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:42897 (local address 127.0.0.1:43070)
0810 14:42:11.344811 (+ 5525us) negotiation.cc:107] Waiting for socket to connect
0810 14:42:11.344825 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:42:11.344936 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:42:11.345825 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:42:11.345832 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:42:11.345872 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:42:11.346235 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:11.346241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.352802 (+ 6561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.352806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:11.353906 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:11.353918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.355275 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.355279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:11.355355 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:11.355372 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 14:42:11.355464 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 14:42:11.356334 (+ 870us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5453,"mutex_wait_us":950,"spinlock_wait_cycles":6784,"thread_start_us":125,"threads_started":1}
I20260810 14:42:11.357369 1504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.336977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:34513 (local address 127.0.0.1:47838)
0810 14:42:11.343764 (+ 6787us) negotiation.cc:107] Waiting for socket to connect
0810 14:42:11.343851 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0810 14:42:11.344021 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:42:11.345825 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:42:11.345832 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:42:11.345873 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:42:11.346060 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:11.346068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.349557 (+ 3489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.349560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:11.351036 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:11.351049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.356937 (+ 5888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.356941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:11.357034 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:11.357058 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:42:11.357136 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:42:11.357241 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6514,"mutex_wait_us":2,"spinlock_wait_cycles":7040,"thread_start_us":869,"threads_started":1}
I20260810 14:42:11.357888 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.340540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.15.84.1:42897)
0810 14:42:11.344394 (+ 3854us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:11.344402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:11.345048 (+ 646us) server_negotiation.cc:407] Connection header received
0810 14:42:11.345189 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:11.345197 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:11.345463 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:11.345625 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:42:11.348053 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.352462 (+ 4409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.354070 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.354930 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.355032 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:11.355205 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:11.357619 (+ 2414us) server_negotiation.cc:299] Negotiation successful
0810 14:42:11.357757 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3724,"thread_start_us":135,"threads_started":1}
I20260810 14:42:11.360354 1123 tablet_service.cc:1467] Processing CreateTablet for tablet 0d58620e0e0c494685e3492565b9484b (DEFAULT_TABLE table=test-table [id=55e15809766b443682e58d2f928e1a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:42:11.360903 900 tablet_service.cc:1467] Processing CreateTablet for tablet 0d58620e0e0c494685e3492565b9484b (DEFAULT_TABLE table=test-table [id=55e15809766b443682e58d2f928e1a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:42:11.361780 1123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d58620e0e0c494685e3492565b9484b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:42:11.362082 900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d58620e0e0c494685e3492565b9484b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:42:11.369223 1501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.337461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.15.84.3:34513)
0810 14:42:11.345110 (+ 7649us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:11.345122 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:11.345142 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:42:11.345234 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:11.345238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:11.345462 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:11.345625 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:42:11.348276 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.349427 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.355702 (+ 6275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.356570 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.356685 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:11.356865 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:11.357254 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 14:42:11.357366 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7528,"thread_start_us":198,"threads_started":1}
I20260810 14:42:11.372848 1345 tablet_service.cc:1467] Processing CreateTablet for tablet 0d58620e0e0c494685e3492565b9484b (DEFAULT_TABLE table=test-table [id=55e15809766b443682e58d2f928e1a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:42:11.374915 1345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d58620e0e0c494685e3492565b9484b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:42:11.389601 1516 tablet_bootstrap.cc:492] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221: Bootstrap starting.
I20260810 14:42:11.393401 1514 tablet_bootstrap.cc:492] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a: Bootstrap starting.
I20260810 14:42:11.396468 1514 tablet_bootstrap.cc:654] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a: Neither blocks nor log segments found. Creating new log.
I20260810 14:42:11.397523 1514 log.cc:824] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:42:11.398551 1509 tablet_bootstrap.cc:492] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e: Bootstrap starting.
I20260810 14:42:11.403434 1516 tablet_bootstrap.cc:654] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221: Neither blocks nor log segments found. Creating new log.
I20260810 14:42:11.404867 1516 log.cc:824] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221: Log is configured to *not* fsync() on all Append() calls
I20260810 14:42:11.405619 1509 tablet_bootstrap.cc:654] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e: Neither blocks nor log segments found. Creating new log.
I20260810 14:42:11.406625 1509 log.cc:824] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:42:11.411095 1514 tablet_bootstrap.cc:492] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a: No bootstrap required, opened a new log
I20260810 14:42:11.411374 1514 ts_tablet_manager.cc:1397] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260810 14:42:11.414675 1514 raft_consensus.cc:348] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4a89dac3e14147bc2a0d46e297406a" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34513 } } peers { permanent_uuid: "6a64301b6bf04c7dbf02e60333f1d61e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42897 } } peers { permanent_uuid: "93f14190ada440d9a6fdcfcec564d221" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35303 } }
I20260810 14:42:11.414938 1514 raft_consensus.cc:374] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:42:11.414991 1514 raft_consensus.cc:729] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4a89dac3e14147bc2a0d46e297406a, State: Initialized, Role: FOLLOWER
I20260810 14:42:11.415547 1514 consensus_queue.cc:260] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4a89dac3e14147bc2a0d46e297406a" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34513 } } peers { permanent_uuid: "6a64301b6bf04c7dbf02e60333f1d61e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42897 } } peers { permanent_uuid: "93f14190ada440d9a6fdcfcec564d221" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35303 } }
I20260810 14:42:11.416702 1516 tablet_bootstrap.cc:492] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221: No bootstrap required, opened a new log
I20260810 14:42:11.416714 1514 ts_tablet_manager.cc:1428] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 14:42:11.416996 1516 ts_tablet_manager.cc:1397] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20260810 14:42:11.417115 1483 heartbeater.cc:503] Master 127.15.84.62:41607 was elected leader, sending a full tablet report...
I20260810 14:42:11.424961 1516 raft_consensus.cc:348] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4a89dac3e14147bc2a0d46e297406a" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34513 } } peers { permanent_uuid: "6a64301b6bf04c7dbf02e60333f1d61e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42897 } } peers { permanent_uuid: "93f14190ada440d9a6fdcfcec564d221" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35303 } }
I20260810 14:42:11.427798 1509 tablet_bootstrap.cc:492] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e: No bootstrap required, opened a new log
I20260810 14:42:11.428069 1509 ts_tablet_manager.cc:1397] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.006s
I20260810 14:42:11.433334 1509 raft_consensus.cc:348] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4a89dac3e14147bc2a0d46e297406a" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34513 } } peers { permanent_uuid: "6a64301b6bf04c7dbf02e60333f1d61e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42897 } } peers { permanent_uuid: "93f14190ada440d9a6fdcfcec564d221" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35303 } }
I20260810 14:42:11.433830 1509 raft_consensus.cc:374] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:42:11.433943 1509 raft_consensus.cc:729] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a64301b6bf04c7dbf02e60333f1d61e, State: Initialized, Role: FOLLOWER
I20260810 14:42:11.434653 1509 consensus_queue.cc:260] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4a89dac3e14147bc2a0d46e297406a" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34513 } } peers { permanent_uuid: "6a64301b6bf04c7dbf02e60333f1d61e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42897 } } peers { permanent_uuid: "93f14190ada440d9a6fdcfcec564d221" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35303 } }
I20260810 14:42:11.436013 1509 ts_tablet_manager.cc:1428] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 14:42:11.436539 1031 heartbeater.cc:503] Master 127.15.84.62:41607 was elected leader, sending a full tablet report...
I20260810 14:42:11.439651 1516 raft_consensus.cc:374] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:42:11.439823 1516 raft_consensus.cc:729] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f14190ada440d9a6fdcfcec564d221, State: Initialized, Role: FOLLOWER
I20260810 14:42:11.440516 1516 consensus_queue.cc:260] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4a89dac3e14147bc2a0d46e297406a" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34513 } } peers { permanent_uuid: "6a64301b6bf04c7dbf02e60333f1d61e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42897 } } peers { permanent_uuid: "93f14190ada440d9a6fdcfcec564d221" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35303 } }
I20260810 14:42:11.442142 1516 ts_tablet_manager.cc:1428] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260810 14:42:11.444569 1262 heartbeater.cc:503] Master 127.15.84.62:41607 was elected leader, sending a full tablet report...
W20260810 14:42:11.504519 1485 tablet.cc:2369] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:42:11.523715 1032 tablet.cc:2369] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:42:11.631385 1265 tablet.cc:2369] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:42:11.836536 1521 raft_consensus.cc:482] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:42:11.836787 1521 raft_consensus.cc:504] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4a89dac3e14147bc2a0d46e297406a" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34513 } } peers { permanent_uuid: "6a64301b6bf04c7dbf02e60333f1d61e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42897 } } peers { permanent_uuid: "93f14190ada440d9a6fdcfcec564d221" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35303 } }
I20260810 14:42:11.838079 1521 leader_election.cc:288] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a64301b6bf04c7dbf02e60333f1d61e (127.15.84.1:42897), 93f14190ada440d9a6fdcfcec564d221 (127.15.84.2:35303)
I20260810 14:42:11.843169 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.838878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:51703 (local address 127.15.84.2:35303)
0810 14:42:11.839006 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:11.839010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:11.839086 (+ 76us) server_negotiation.cc:407] Connection header received
0810 14:42:11.839210 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:11.839213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:11.839256 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:11.839323 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:42:11.839571 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.840320 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.841908 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.842517 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.842622 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:11.842692 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:11.842942 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 14:42:11.843017 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:42:11.843153 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.838459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:52819 (local address 127.15.84.1:42897)
0810 14:42:11.838593 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:11.838597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:11.838687 (+ 90us) server_negotiation.cc:407] Connection header received
0810 14:42:11.838931 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:11.838935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:11.838995 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:11.839086 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:42:11.839391 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.840206 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.841908 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.842516 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.842622 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:11.842708 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:11.842942 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 14:42:11.843017 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:42:11.843238 1542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.838750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:35303 (local address 127.15.84.3:51703)
0810 14:42:11.838997 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 14:42:11.839008 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:42:11.839124 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:42:11.839355 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:42:11.839358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:42:11.839376 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:42:11.839476 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:11.839481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.840410 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.840413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:11.841768 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:11.841774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.842634 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.842638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:11.842783 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:11.842842 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 14:42:11.842917 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 14:42:11.843056 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":143,"thread_start_us":86,"threads_started":1}
I20260810 14:42:11.843237 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.838346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:42897 (local address 127.15.84.3:52819)
0810 14:42:11.838595 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 14:42:11.838610 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:42:11.838735 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:42:11.839121 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:42:11.839124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:42:11.839138 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:42:11.839267 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:11.839273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.840306 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.840309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:11.841761 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:11.841774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.842634 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.842638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:11.842780 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:11.842840 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 14:42:11.842912 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 14:42:11.843056 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":116,"thread_start_us":86,"threads_started":1}
I20260810 14:42:11.843966 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d58620e0e0c494685e3492565b9484b" candidate_uuid: "6b4a89dac3e14147bc2a0d46e297406a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a64301b6bf04c7dbf02e60333f1d61e" is_pre_election: true
I20260810 14:42:11.843978 1176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d58620e0e0c494685e3492565b9484b" candidate_uuid: "6b4a89dac3e14147bc2a0d46e297406a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f14190ada440d9a6fdcfcec564d221" is_pre_election: true
I20260810 14:42:11.844274 924 raft_consensus.cc:2454] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b4a89dac3e14147bc2a0d46e297406a in term 0.
I20260810 14:42:11.844292 1176 raft_consensus.cc:2454] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b4a89dac3e14147bc2a0d46e297406a in term 0.
I20260810 14:42:11.844779 1302 leader_election.cc:302] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a64301b6bf04c7dbf02e60333f1d61e, 6b4a89dac3e14147bc2a0d46e297406a; no voters:
I20260810 14:42:11.845067 1521 raft_consensus.cc:2781] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:42:11.845176 1521 raft_consensus.cc:482] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:42:11.845233 1521 raft_consensus.cc:3037] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:42:11.846359 1521 raft_consensus.cc:504] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4a89dac3e14147bc2a0d46e297406a" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34513 } } peers { permanent_uuid: "6a64301b6bf04c7dbf02e60333f1d61e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42897 } } peers { permanent_uuid: "93f14190ada440d9a6fdcfcec564d221" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35303 } }
I20260810 14:42:11.846712 1521 leader_election.cc:288] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [CANDIDATE]: Term 1 election: Requested vote from peers 6a64301b6bf04c7dbf02e60333f1d61e (127.15.84.1:42897), 93f14190ada440d9a6fdcfcec564d221 (127.15.84.2:35303)
I20260810 14:42:11.847080 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d58620e0e0c494685e3492565b9484b" candidate_uuid: "6b4a89dac3e14147bc2a0d46e297406a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a64301b6bf04c7dbf02e60333f1d61e"
I20260810 14:42:11.847141 1176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d58620e0e0c494685e3492565b9484b" candidate_uuid: "6b4a89dac3e14147bc2a0d46e297406a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f14190ada440d9a6fdcfcec564d221"
I20260810 14:42:11.847229 924 raft_consensus.cc:3037] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:42:11.847266 1176 raft_consensus.cc:3037] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:42:11.848572 1176 raft_consensus.cc:2454] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b4a89dac3e14147bc2a0d46e297406a in term 1.
I20260810 14:42:11.848579 924 raft_consensus.cc:2454] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b4a89dac3e14147bc2a0d46e297406a in term 1.
I20260810 14:42:11.848909 1302 leader_election.cc:302] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a64301b6bf04c7dbf02e60333f1d61e, 6b4a89dac3e14147bc2a0d46e297406a; no voters:
I20260810 14:42:11.849088 1521 raft_consensus.cc:2781] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:42:11.849351 1521 raft_consensus.cc:686] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [term 1 LEADER]: Becoming Leader. State: Replica: 6b4a89dac3e14147bc2a0d46e297406a, State: Running, Role: LEADER
I20260810 14:42:11.849756 1521 consensus_queue.cc:237] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4a89dac3e14147bc2a0d46e297406a" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34513 } } peers { permanent_uuid: "6a64301b6bf04c7dbf02e60333f1d61e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42897 } } peers { permanent_uuid: "93f14190ada440d9a6fdcfcec564d221" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35303 } }
I20260810 14:42:11.854167 683 catalog_manager.cc:5495] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b4a89dac3e14147bc2a0d46e297406a (127.15.84.3). New cstate: current_term: 1 leader_uuid: "6b4a89dac3e14147bc2a0d46e297406a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4a89dac3e14147bc2a0d46e297406a" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 34513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a64301b6bf04c7dbf02e60333f1d61e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f14190ada440d9a6fdcfcec564d221" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35303 } health_report { overall_health: UNKNOWN } } }
I20260810 14:42:11.893584 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.889370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.15.84.3:34513)
0810 14:42:11.889708 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:11.889713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:11.889731 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:42:11.889912 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:11.889915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:11.889988 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:11.890118 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:42:11.890484 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.891246 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.892119 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:11.892362 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:11.892439 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:11.892679 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:42:11.893139 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:42:11.893313 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:11.893363 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 14:42:11.893422 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":99,"threads_started":1}
I20260810 14:42:11.900058 1176 raft_consensus.cc:1261] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221 [term 1 FOLLOWER]: Refusing update from remote peer 6b4a89dac3e14147bc2a0d46e297406a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:42:11.900058 924 raft_consensus.cc:1261] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e [term 1 FOLLOWER]: Refusing update from remote peer 6b4a89dac3e14147bc2a0d46e297406a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:42:11.900786 1545 consensus_queue.cc:1035] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f14190ada440d9a6fdcfcec564d221" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 35303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:42:11.900971 1521 consensus_queue.cc:1035] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a64301b6bf04c7dbf02e60333f1d61e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:42:11.918699 683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:42:11.918912 683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:42:11.921396 683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 547dfc791cb248cfb318c1c3df2ca087: Sending DeleteTablet for 3 replicas of tablet 0d58620e0e0c494685e3492565b9484b
I20260810 14:42:11.922237 1345 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d58620e0e0c494685e3492565b9484b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:42:11 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:47838
I20260810 14:42:11.922264 1123 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d58620e0e0c494685e3492565b9484b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:42:11 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:42366
I20260810 14:42:11.922339 900 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d58620e0e0c494685e3492565b9484b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:42:11 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:43070
I20260810 14:42:11.922839 1576 tablet_replica.cc:330] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a: stopping tablet replica
I20260810 14:42:11.922935 1577 tablet_replica.cc:330] T 0d58620e0e0c494685e3492565b9484b P 6a64301b6bf04c7dbf02e60333f1d61e: stopping tablet replica
I20260810 14:42:11.923329 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 814
I20260810 14:42:11.948112 1578 tablet_replica.cc:330] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221: stopping tablet replica
I20260810 14:42:11.948565 1578 raft_consensus.cc:2229] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:42:11.948967 1578 raft_consensus.cc:2258] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:42:11.951951 1578 ts_tablet_manager.cc:1905] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:42:11.954500 1578 ts_tablet_manager.cc:1918] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:42:11.954594 1578 log.cc:1196] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/0d58620e0e0c494685e3492565b9484b
I20260810 14:42:11.954847 1578 ts_tablet_manager.cc:1939] T 0d58620e0e0c494685e3492565b9484b P 93f14190ada440d9a6fdcfcec564d221: Deleting consensus metadata
I20260810 14:42:11.956322 650 catalog_manager.cc:4841] TS 93f14190ada440d9a6fdcfcec564d221 (127.15.84.2:35303): tablet 0d58620e0e0c494685e3492565b9484b (table test-table [id=55e15809766b443682e58d2f928e1a3b]) successfully deleted
I20260810 14:42:11.959754 1576 raft_consensus.cc:2229] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:42:11.960240 1576 raft_consensus.cc:2258] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:42:11.961901 1576 ts_tablet_manager.cc:1905] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:42:11.964844 1576 ts_tablet_manager.cc:1918] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:42:11.964955 1576 log.cc:1196] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/0d58620e0e0c494685e3492565b9484b
I20260810 14:42:11.965306 1576 ts_tablet_manager.cc:1939] T 0d58620e0e0c494685e3492565b9484b P 6b4a89dac3e14147bc2a0d46e297406a: Deleting consensus metadata
W20260810 14:42:11.965858 650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:42897 (error 108)
I20260810 14:42:11.966596 652 catalog_manager.cc:4841] TS 6b4a89dac3e14147bc2a0d46e297406a (127.15.84.3:34513): tablet 0d58620e0e0c494685e3492565b9484b (table test-table [id=55e15809766b443682e58d2f928e1a3b]) successfully deleted
I20260810 14:42:11.967084 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:11.966605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:42897 (local address 127.0.0.1:43076)
0810 14:42:11.966890 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 14:42:11.966976 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:42897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":64,"threads_started":1}
I20260810 14:42:11.967127 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 1040
W20260810 14:42:11.974100 650 catalog_manager.cc:4584] TS 6a64301b6bf04c7dbf02e60333f1d61e (127.15.84.1:42897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d58620e0e0c494685e3492565b9484b: Network error: Client connection negotiation failed: client connection to 127.15.84.1:42897: connect: Connection refused (error 111)
I20260810 14:42:11.984493 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 1276
I20260810 14:42:12.000653 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 625
I20260810 14:42:12.019749 15696 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 14:42:12.366598 15696 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 14:42:12.366746 15696 mini_ranger.cc:66] Stopping Ranger...
I20260810 14:42:12.711961 15696 mini_ranger.cc:68] Stopped Ranger
2026-08-10 14:42:12.712 UTC [32432] LOG: received smart shutdown request
2026-08-10 14:42:12.716 UTC [32432] LOG: background worker "logical replication launcher" (PID 32442) exited with exit code 1
2026-08-10 14:42:12.716 UTC [32437] LOG: shutting down
2026-08-10 14:42:12.867 UTC [32432] LOG: database system is shut down
2026-08-10T14:42:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47513 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:42:13 dist-test-slave-rmhg krb5kdc[1592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:42:13 dist-test-slave-rmhg krb5kdc[1592](info): set up 2 sockets
Aug 10 14:42:13 dist-test-slave-rmhg krb5kdc[1592](info): commencing operation
krb5kdc: starting...
W20260810 14:42:14.108415 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:42:14 dist-test-slave-rmhg krb5kdc[1592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:42:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:42:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/rangeradmin_127.15.84.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/rangeradmin_127.15.84.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/rangerlookup_127.15.84.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/rangerlookup_127.15.84.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/HTTP_127.15.84.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/HTTP_127.15.84.20@KRBTEST.COM.keytab.
I20260810 14:42:14.287539 15696 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPiaUUw/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/postgres -l logfile start
2026-08-10 14:42:16.687 UTC [1644] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 14:42:16.687 UTC [1644] LOG: listening on IPv4 address "127.15.84.20", port 37555
2026-08-10 14:42:16.691 UTC [1644] LOG: listening on Unix socket "/tmp/.s.PGSQL.37555"
2026-08-10 14:42:16.720 UTC [1649] LOG: database system was shut down at 2026-08-10 14:42:15 UTC
2026-08-10 14:42:16.724 UTC [1644] LOG: database system is ready to accept connections
I20260810 14:42:17.784139 15696 mini_postgres.cc:96] Postgres bound to 37555
2026-08-10 14:42:17.791 UTC [1662] FATAL: database "slave" does not exist
127.15.84.20:37555 - accepting connections
I20260810 14:42:17.792258 15696 mini_ranger.cc:161] Starting Ranger...
I20260810 14:42:17.807953 15696 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 14:42:18.269086 15696 mini_ranger.cc:87] Created ranger Postgres database
I20260810 14:42:18.269212 15696 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-admin
2026-08-10 14:42:18,551 [I] DB FLAVOR :POSTGRES
2026-08-10 14:42:18,551 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:42:18,551 [I] Checking connection..
2026-08-10 14:42:18,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 14:42:18,824 [I] Checking connection passed.
2026-08-10 14:42:18,824 [I] --------- Verifying version history table ---------
2026-08-10 14:42:18,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:42:19,077 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:42:19,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:42:19,339 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:42:19,339 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 14:42:19,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 14:42:19.605 UTC [1773] WARNING: there is no transaction in progress
2026-08-10 14:42:19,614 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 14:42:19,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:42:19,868 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 14:42:19,868 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:42:19,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 14:42:20,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 14:42:20,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rmhg.c.gcp-upstream.internal','N') ;"
2026-08-10 14:42:20,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:42:20,911 [I] Table x_portal_user does not exist in database ranger
2026-08-10 14:42:20,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:42:21,155 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 14:42:21,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:42:21,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:42:21,696 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 14:42:21,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 14:42:22.685 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.703 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.725 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.741 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.769 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.788 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.848 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.854 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.868 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.879 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.889 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.903 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.913 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.919 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.925 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:22.931 UTC [1944] WARNING: there is no transaction in progress
2026-08-10 14:42:23,557 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 14:42:23,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 14:42:23,806 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 14:42:23,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:42:24,064 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 14:42:24,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:42:24,311 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 14:42:24,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:42:24,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:42:24,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rmhg.c.gcp-upstream.internal';"
2026-08-10 14:42:25,088 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 14:42:25,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 14:42:25,357 [I] DB_PATCHES have already been applied
I20260810 14:42:25.363519 15696 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPiaUUw/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 14:42:25.363593 15696 mini_ranger.cc:193] Using host: 127.15.84.20
I20260810 14:42:25.368180 15696 mini_ranger.cc:227] Ranger admin URL: http://127.15.84.20:48059
Aug 10, 2026 2:42:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:42:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:42:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 2:42:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 2:42:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 2:42:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 2:42:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 14:42:26.610373 15696 mini_ranger.cc:160] Time spent starting Ranger: real 8.818s user 0.000s sys 0.012s
Aug 10, 2026 2:42:47 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 2:42:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 2:42:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 2:42:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 2:42:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 2:42:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 2:42:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 2:42:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 2:42:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 2:42:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 2:42:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 2:42:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 2:42:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 2:42:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 2:42:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 2:42:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 2:42:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 2:42:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 14:42:49.133347 15696 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/rangerkms_127.15.84.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/rangerkms_127.15.84.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.84.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.84.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.84.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/HTTP_127.15.84.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.84.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/HTTP_127.15.84.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 14:42:49 dist-test-slave-rmhg krb5kdc[1592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372969, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 14:42:49.218784 15696 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 14:42:49.229809 15696 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 14:42:50.032812 15696 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 14:42:50.032893 15696 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-kms
I20260810 14:42:50.032913 15696 mini_ranger_kms.cc:227] Using postgres at 127.15.84.20:37555
2026-08-10 14:42:50,216 [I] DB FLAVOR :POSTGRES
2026-08-10 14:42:50,216 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:42:50,216 [I] Checking connection
2026-08-10 14:42:50,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 14:42:50,476 [I] connection success
2026-08-10 14:42:50,476 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 14:42:50,477 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 14:42:50,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 14:42:50,727 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 14:42:50,727 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:42:50,727 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 14:42:50,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.84.20:37555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 14:42:50,989 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 14:42:51.468442 15696 mini_ranger_kms.cc:310] Created kms service
I20260810 14:42:51.680915 15696 mini_ranger_kms.cc:326] Created kudu user
I20260810 14:42:51.731437 15696 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 14:42:51 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372969, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:42:52.122596 15696 mini_ranger_kms.cc:384] Added ranger policy
I20260810 14:42:52.122853 15696 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPiaUUw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPiaUUw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 14:42:52.122875 15696 mini_ranger_kms.cc:240] Using host: 127.15.84.20
I20260810 14:42:52.128816 15696 mini_ranger_kms.cc:276] Ranger KMS PID: 2223
I20260810 14:42:52.128906 15696 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.84.20:43847
Aug 10, 2026 2:42:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:42:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:42:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 2:42:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 2:42:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 2:42:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 14:42:54 dist-test-slave-rmhg krb5kdc[1592](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372974, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.84.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:42:54.641571 15696 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.423s user 0.028s sys 0.139s
I20260810 14:42:54.641700 15696 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 14:42:54.641777 15696 mini_ranger_kms.cc:401] 127.15.84.20:43847/kms/v1/keys
Aug 10 14:42:54 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372974, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.84.20@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
Aug 10 14:42:55 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372974, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.84.20@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
Aug 10 14:42:57 dist-test-slave-rmhg krb5kdc[1592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:42:57.634506 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:39955
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:39955
--ranger_config_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.84.20:43847/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:42:57.757444 2310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:42:57.757763 2310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:42:57.757804 2310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:42:57.762010 2310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:42:57.762082 2310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:42:57.762100 2310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:42:57.762120 2310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:42:57.762148 2310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:42:57.767912 2310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.84.20:43847/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:39955
--ranger_config_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:39955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:42:57.769107 2310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:42:57.770223 2310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:42:57.779757 2310 server_base.cc:1034] running on GCE node
W20260810 14:42:57.779713 2318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:57.779722 2320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:57.779711 2322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:42:57.780701 2310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:42:57.782094 2310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:42:57.783277 2310 hybrid_clock.cc:648] HybridClock initialized: now 1786372977783245 us; error 53 us; skew 500 ppm
Aug 10 14:42:57 dist-test-slave-rmhg krb5kdc[1592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372977, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:42:57.797700 2310 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:42:57.798878 2310 webserver.cc:460] Webserver started at http://127.15.84.62:34789/ using document root <none> and password file <none>
I20260810 14:42:57.799510 2310 fs_manager.cc:359] Metadata directory not provided
I20260810 14:42:57.799571 2310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:42:57.799760 2310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:42:57 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372977, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:42:57.858946 2310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "dc7301a291964706b44b5f7683788274"
format_stamp: "Formatted at 2026-08-10 14:42:57 on dist-test-slave-rmhg"
server_key: "4d38683fd4e6896b32b23f377b5348ce"
server_key_iv: "b04b20ab0ed45c4844704f94f79fd13f"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:57.859784 2310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "dc7301a291964706b44b5f7683788274"
format_stamp: "Formatted at 2026-08-10 14:42:57 on dist-test-slave-rmhg"
server_key: "4d38683fd4e6896b32b23f377b5348ce"
server_key_iv: "b04b20ab0ed45c4844704f94f79fd13f"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:57.865254 2310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 14:42:57.880795 2332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:57.882344 2310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:42:57.882488 2310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "dc7301a291964706b44b5f7683788274"
format_stamp: "Formatted at 2026-08-10 14:42:57 on dist-test-slave-rmhg"
server_key: "4d38683fd4e6896b32b23f377b5348ce"
server_key_iv: "b04b20ab0ed45c4844704f94f79fd13f"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:57.882634 2310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:42:57.917979 2310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:42:57.920939 2310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:42:57.921141 2310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:42:57.929440 2310 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:39955
I20260810 14:42:57.929483 2411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:39955 every 8 connection(s)
I20260810 14:42:57.930516 2310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:42:57.934257 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:42:57.934397 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 2310
I20260810 14:42:57.934518 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:42:57.934830 15696 external_mini_cluster.cc:1428] Setting key 67124215fecca3411898151d517962e4
I20260810 14:42:57.942513 2412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274: Bootstrap starting.
Aug 10 14:42:57 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:42:57.945537 2412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274: Neither blocks nor log segments found. Creating new log.
I20260810 14:42:57.946936 2412 log.cc:824] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274: Log is configured to *not* fsync() on all Append() calls
I20260810 14:42:57.949424 2412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274: No bootstrap required, opened a new log
I20260810 14:42:57.949704 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:57.936277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51648 (local address 127.15.84.62:39955)
0810 14:42:57.936730 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:57.936752 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:57.936983 (+ 231us) server_negotiation.cc:407] Connection header received
0810 14:42:57.937958 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:57.937980 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:57.938364 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:57.938699 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:42:57.939526 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:57.940576 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:57.941531 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:57.942051 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:57.942208 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:57.944708 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:42:57.944714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:42:57.944727 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:42:57.944759 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:42:57.946317 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:57.946744 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:57.946749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:57.946754 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:57.946864 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:57.947171 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:57.947174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:57.947175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:57.947689 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:42:57.947902 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:57.948229 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 14:42:57.948697 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":124,"threads_started":1}
I20260810 14:42:57.952921 2412 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7301a291964706b44b5f7683788274" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39955 } }
I20260810 14:42:57.953177 2412 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:42:57.953241 2412 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7301a291964706b44b5f7683788274, State: Initialized, Role: FOLLOWER
I20260810 14:42:57.953678 2412 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7301a291964706b44b5f7683788274" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39955 } }
I20260810 14:42:57.953804 2412 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:42:57.953866 2412 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:42:57.953960 2412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:42:57.954873 2412 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7301a291964706b44b5f7683788274" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39955 } }
I20260810 14:42:57.955214 2412 leader_election.cc:302] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc7301a291964706b44b5f7683788274; no voters:
I20260810 14:42:57.955521 2412 leader_election.cc:288] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:42:57.955639 2422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:42:57.955861 2422 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 1 LEADER]: Becoming Leader. State: Replica: dc7301a291964706b44b5f7683788274, State: Running, Role: LEADER
I20260810 14:42:57.956254 2422 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7301a291964706b44b5f7683788274" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39955 } }
I20260810 14:42:57.956822 2412 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:42:57.957643 2422 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc7301a291964706b44b5f7683788274. Latest consensus state: current_term: 1 leader_uuid: "dc7301a291964706b44b5f7683788274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7301a291964706b44b5f7683788274" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39955 } } }
I20260810 14:42:57.957863 2422 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [sys.catalog]: This master's current role is: LEADER
I20260810 14:42:57.958089 2424 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc7301a291964706b44b5f7683788274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7301a291964706b44b5f7683788274" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39955 } } }
I20260810 14:42:57.958189 2424 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [sys.catalog]: This master's current role is: LEADER
I20260810 14:42:57.958302 2435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:42:57.961716 2435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:42:57.962148 2412 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 14:42:57.973429 2435 catalog_manager.cc:1347] Generated new cluster ID: e2d7f2740d1d44a28a0d42d0696eb033
I20260810 14:42:57.973532 2435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:42:57.995500 2435 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:42:57.997005 2435 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:42:58.013456 2435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274: Generated new TSK 0
I20260810 14:42:58.015252 2435 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:42:58.770571 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.84.20:43847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39955
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:42:58.900487 2489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:42:58.901006 2489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:42:58.901129 2489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:42:58.906391 2489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:42:58.906476 2489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:42:58.906559 2489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:42:58.912352 2489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.84.20:43847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:42:58.913647 2489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:42:58.914763 2489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:42:58.925017 2489 server_base.cc:1034] running on GCE node
W20260810 14:42:58.925040 2506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:58.924999 2505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:58.925311 2508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:42:58.927793 2489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:42:58.928752 2489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:42:58.929965 2489 hybrid_clock.cc:648] HybridClock initialized: now 1786372978929930 us; error 51 us; skew 500 ppm
Aug 10 14:42:58 dist-test-slave-rmhg krb5kdc[1592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372978, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:42:58.933836 2489 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:42:58.935097 2489 webserver.cc:460] Webserver started at http://127.15.84.1:39739/ using document root <none> and password file <none>
I20260810 14:42:58.935915 2489 fs_manager.cc:359] Metadata directory not provided
I20260810 14:42:58.936080 2489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:42:58.936394 2489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:42:58 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372978, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:42:58.956350 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "d47deef295c34e7193d914d13b120332"
format_stamp: "Formatted at 2026-08-10 14:42:58 on dist-test-slave-rmhg"
server_key: "faff7aa0f93f22b70823b9a84a46235c"
server_key_iv: "feddb8bc66d8932ac60eeef3364ece09"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:58.957267 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "d47deef295c34e7193d914d13b120332"
format_stamp: "Formatted at 2026-08-10 14:42:58 on dist-test-slave-rmhg"
server_key: "faff7aa0f93f22b70823b9a84a46235c"
server_key_iv: "feddb8bc66d8932ac60eeef3364ece09"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:58.963167 2489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 14:42:58.990466 2516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:58.994567 2489 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.001s
I20260810 14:42:58.994751 2489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "d47deef295c34e7193d914d13b120332"
format_stamp: "Formatted at 2026-08-10 14:42:58 on dist-test-slave-rmhg"
server_key: "faff7aa0f93f22b70823b9a84a46235c"
server_key_iv: "feddb8bc66d8932ac60eeef3364ece09"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:58.994907 2489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:42:59.012670 2489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:42:59.016872 2489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:42:59.017071 2489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:42:59.017724 2489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:42:59.019744 2489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:42:59.019812 2489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:59.019873 2489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:42:59.019917 2489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:59.035313 2711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:42817 every 8 connection(s)
I20260810 14:42:59.035656 2489 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:42817
I20260810 14:42:59.036969 2489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:42:59.040241 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 2489
I20260810 14:42:59.040373 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:42:59.040714 15696 external_mini_cluster.cc:1428] Setting key d0d5508ad315089d22099382606c0976
Aug 10 14:42:59 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372978, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:42:59.060468 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:59.041457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:54103 (local address 127.15.84.62:39955)
0810 14:42:59.042054 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:59.042058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:59.043868 (+ 1810us) server_negotiation.cc:407] Connection header received
0810 14:42:59.043920 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:59.043924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:59.043980 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:59.044064 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:42:59.045830 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:59.046721 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:59.048508 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:59.048881 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:59.048954 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:59.055557 (+ 6603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:42:59.055562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:42:59.055565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:42:59.055596 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:42:59.057547 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:59.058316 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:59.058320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:59.058322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:59.058432 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:59.058942 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:59.058946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:59.058948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:59.059215 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:42:59.059324 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:59.060132 (+ 808us) server_negotiation.cc:299] Negotiation successful
0810 14:42:59.060279 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
I20260810 14:42:59.061601 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:59.040278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:39955 (local address 127.15.84.1:54103)
0810 14:42:59.042351 (+ 2073us) negotiation.cc:107] Waiting for socket to connect
0810 14:42:59.042420 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 14:42:59.043516 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:42:59.044408 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:42:59.044418 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:42:59.044842 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:42:59.045647 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:59.045669 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:59.046884 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:59.046887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:59.048366 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:59.048378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:59.049205 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:59.049209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:59.049423 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:59.053420 (+ 3997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:42:59.053446 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:42:59.055289 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:42:59.057737 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:42:59.057744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:42:59.057766 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:42:59.058176 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:42:59.058558 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:42:59.058561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:42:59.058563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:42:59.058809 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:42:59.059352 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:42:59.059360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:42:59.059794 (+ 434us) client_negotiation.cc:769] Sending connection context
0810 14:42:59.060038 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 14:42:59.060416 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1824,"thread_start_us":143,"threads_started":1}
I20260810 14:42:59.063766 2712 heartbeater.cc:347] Connected to a master server at 127.15.84.62:39955
I20260810 14:42:59.064046 2712 heartbeater.cc:464] Registering TS with master...
I20260810 14:42:59.064697 2712 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
I20260810 14:42:59.066632 2364 ts_manager.cc:194] Registered new tserver with Master: d47deef295c34e7193d914d13b120332 (127.15.84.1:42817)
I20260810 14:42:59.068259 2364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:54103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:42:59.084815 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.84.20:43847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39955
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:42:59 dist-test-slave-rmhg krb5kdc[1592](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372979, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 14:42:59.249541 2724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:42:59.249982 2724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:42:59.250119 2724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:42:59.256357 2724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:42:59.256502 2724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:42:59.256613 2724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:42:59.264822 2724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.84.20:43847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:42:59.266526 2724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:42:59.267880 2724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:42:59.280663 2734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:59.280674 2739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:59.281087 2735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:42:59.281623 2724 server_base.cc:1034] running on GCE node
I20260810 14:42:59.282338 2724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:42:59.283264 2724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:42:59.284461 2724 hybrid_clock.cc:648] HybridClock initialized: now 1786372979284429 us; error 64 us; skew 500 ppm
Aug 10 14:42:59 dist-test-slave-rmhg krb5kdc[1592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372979, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:42:59.288578 2724 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:42:59.289974 2724 webserver.cc:460] Webserver started at http://127.15.84.2:44761/ using document root <none> and password file <none>
I20260810 14:42:59.290802 2724 fs_manager.cc:359] Metadata directory not provided
I20260810 14:42:59.290911 2724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:42:59.291260 2724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:42:59 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372979, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:42:59.308626 2724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "68c85b5b33dd45cfb629d8ced69adb48"
format_stamp: "Formatted at 2026-08-10 14:42:59 on dist-test-slave-rmhg"
server_key: "96f3f37917402f4a675fbe4b41166bb1"
server_key_iv: "daa4ff280bf439621cc92fbd01d8b127"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:59.309432 2724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "68c85b5b33dd45cfb629d8ced69adb48"
format_stamp: "Formatted at 2026-08-10 14:42:59 on dist-test-slave-rmhg"
server_key: "96f3f37917402f4a675fbe4b41166bb1"
server_key_iv: "daa4ff280bf439621cc92fbd01d8b127"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:59.316293 2724 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260810 14:42:59.339475 2757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:59.341549 2724 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 14:42:59.341825 2724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "68c85b5b33dd45cfb629d8ced69adb48"
format_stamp: "Formatted at 2026-08-10 14:42:59 on dist-test-slave-rmhg"
server_key: "96f3f37917402f4a675fbe4b41166bb1"
server_key_iv: "daa4ff280bf439621cc92fbd01d8b127"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:59.342067 2724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:42:59.360242 2724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:42:59.364413 2724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:42:59.364727 2724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:42:59.365655 2724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:42:59.366874 2724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:42:59.370188 2724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260810 14:42:59.370360 2724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:42:59.370421 2724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:59.381814 2724 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:44103
I20260810 14:42:59.383118 2724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:42:59.386346 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 2724
I20260810 14:42:59.386464 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:42:59.386777 15696 external_mini_cluster.cc:1428] Setting key bcd9d9533d6a05604d7594616b3c419b
I20260810 14:42:59.394192 2939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:44103 every 8 connection(s)
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
Aug 10 14:42:59 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372979, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:42:59.427064 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:59.396992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:54119 (local address 127.15.84.62:39955)
0810 14:42:59.397131 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:59.397135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:59.409729 (+ 12594us) server_negotiation.cc:407] Connection header received
0810 14:42:59.410865 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:59.410869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:59.410936 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:59.411016 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:42:59.412789 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:59.413738 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:59.415706 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:59.416092 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:59.416173 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:59.422379 (+ 6206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:42:59.422384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:42:59.422387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:42:59.422424 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:42:59.424236 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:59.425018 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:59.425023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:59.425025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:59.425120 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:59.425598 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:59.425602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:59.425604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:59.425864 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:42:59.425937 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:59.426730 (+ 793us) server_negotiation.cc:299] Negotiation successful
0810 14:42:59.426885 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:42:59.429402 2944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:59.397617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:39955 (local address 127.15.84.2:54119)
0810 14:42:59.409452 (+ 11835us) negotiation.cc:107] Waiting for socket to connect
0810 14:42:59.409538 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0810 14:42:59.410606 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:42:59.411387 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:42:59.411398 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:42:59.411801 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:42:59.412579 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:59.412601 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:59.413913 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:59.413917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:59.415388 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:59.415400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:59.416397 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:59.416400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:59.416620 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:59.420152 (+ 3532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:42:59.420179 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:42:59.422109 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:42:59.424419 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:42:59.424426 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:42:59.424447 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:42:59.424882 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:42:59.425270 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:42:59.425273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:42:59.425275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:42:59.425483 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:42:59.426023 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:42:59.426031 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:42:59.426418 (+ 387us) client_negotiation.cc:769] Sending connection context
0810 14:42:59.427733 (+ 1315us) client_negotiation.cc:240] Negotiation successful
0810 14:42:59.428128 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11471,"thread_start_us":158,"threads_started":1}
I20260810 14:42:59.431231 2941 heartbeater.cc:347] Connected to a master server at 127.15.84.62:39955
I20260810 14:42:59.431514 2941 heartbeater.cc:464] Registering TS with master...
I20260810 14:42:59.432185 2941 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
I20260810 14:42:59.433583 2364 ts_manager.cc:194] Registered new tserver with Master: 68c85b5b33dd45cfb629d8ced69adb48 (127.15.84.2:44103)
I20260810 14:42:59.434368 2364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:54119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:42:59.458534 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.84.20:43847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39955
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:42:59 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372979, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
W20260810 14:42:59.636932 2947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:42:59.637498 2947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:42:59.637614 2947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:42:59.643965 2947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:42:59.644155 2947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:42:59.644291 2947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:42:59.652721 2947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.84.20:43847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:42:59.654253 2947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:42:59.655709 2947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:42:59.668958 2954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:59.669181 2953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:42:59.669608 2957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:42:59.670226 2947 server_base.cc:1034] running on GCE node
I20260810 14:42:59.670773 2947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:42:59.671630 2947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:42:59.672832 2947 hybrid_clock.cc:648] HybridClock initialized: now 1786372979672790 us; error 61 us; skew 500 ppm
Aug 10 14:42:59 dist-test-slave-rmhg krb5kdc[1592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372979, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:42:59.676662 2947 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:42:59.677997 2947 webserver.cc:460] Webserver started at http://127.15.84.3:34979/ using document root <none> and password file <none>
I20260810 14:42:59.678802 2947 fs_manager.cc:359] Metadata directory not provided
I20260810 14:42:59.678954 2947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:42:59.679253 2947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:42:59 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372979, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:42:59.696751 2947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "3ff3156de3344ed0a98795960e3230f6"
format_stamp: "Formatted at 2026-08-10 14:42:59 on dist-test-slave-rmhg"
server_key: "281a5b1d15427cd994bd9b3f02b379f3"
server_key_iv: "d21a8679d73313d4913910f7fe933ab6"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:59.697647 2947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "3ff3156de3344ed0a98795960e3230f6"
format_stamp: "Formatted at 2026-08-10 14:42:59 on dist-test-slave-rmhg"
server_key: "281a5b1d15427cd994bd9b3f02b379f3"
server_key_iv: "d21a8679d73313d4913910f7fe933ab6"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:59.704708 2947 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260810 14:42:59.729617 2967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:59.731513 2947 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:42:59.731746 2947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "3ff3156de3344ed0a98795960e3230f6"
format_stamp: "Formatted at 2026-08-10 14:42:59 on dist-test-slave-rmhg"
server_key: "281a5b1d15427cd994bd9b3f02b379f3"
server_key_iv: "d21a8679d73313d4913910f7fe933ab6"
server_key_version: "kuduclusterkey@0"
I20260810 14:42:59.731966 2947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:42:59.768983 2947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:42:59.773126 2947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:42:59.773494 2947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:42:59.774411 2947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:42:59.775650 2947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:42:59.775768 2947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:59.775880 2947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:42:59.775947 2947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:42:59.787943 2947 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:42609
I20260810 14:42:59.789434 2947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:42:59.789716 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 2947
I20260810 14:42:59.789852 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:42:59.790181 15696 external_mini_cluster.cc:1428] Setting key 023071373f6856f3be97b115289953d9
I20260810 14:42:59.791476 3148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:42609 every 8 connection(s)
Aug 10 14:42:59 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372979, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:42:59.817699 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:59.795804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:36951 (local address 127.15.84.62:39955)
0810 14:42:59.795934 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:42:59.795938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:42:59.798354 (+ 2416us) server_negotiation.cc:407] Connection header received
0810 14:42:59.799755 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:42:59.799760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:42:59.799832 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:42:59.799942 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:42:59.802197 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:59.803060 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:59.805014 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:42:59.805411 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:42:59.805503 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:59.811107 (+ 5604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:42:59.811113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:42:59.811116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:42:59.811164 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:42:59.812884 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:59.813714 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:59.813719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:59.813720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:59.813823 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:42:59.814287 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:42:59.814291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:42:59.814293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:42:59.814539 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:42:59.814615 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:42:59.817347 (+ 2732us) server_negotiation.cc:299] Negotiation successful
0810 14:42:59.817528 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:42:59.818882 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:42:59.794444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:39955 (local address 127.15.84.3:36951)
0810 14:42:59.798096 (+ 3652us) negotiation.cc:107] Waiting for socket to connect
0810 14:42:59.798203 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0810 14:42:59.799438 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:42:59.800432 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:42:59.800444 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:42:59.800963 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:42:59.801963 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:59.801989 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:59.803279 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:59.803283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:59.804840 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:42:59.804856 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:42:59.805757 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:42:59.805761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:42:59.806011 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:42:59.808660 (+ 2649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:42:59.808691 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:42:59.810799 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:42:59.813083 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:42:59.813092 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:42:59.813116 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:42:59.813575 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:42:59.813940 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:42:59.813942 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:42:59.813944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:42:59.814174 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:42:59.814684 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:42:59.814691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:42:59.816921 (+ 2230us) client_negotiation.cc:769] Sending connection context
0810 14:42:59.817228 (+ 307us) client_negotiation.cc:240] Negotiation successful
0810 14:42:59.817654 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3336,"thread_start_us":203,"threads_started":1}
I20260810 14:42:59.820815 3149 heartbeater.cc:347] Connected to a master server at 127.15.84.62:39955
I20260810 14:42:59.821133 3149 heartbeater.cc:464] Registering TS with master...
I20260810 14:42:59.821858 3149 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
I20260810 14:42:59.823287 2364 ts_manager.cc:194] Registered new tserver with Master: 3ff3156de3344ed0a98795960e3230f6 (127.15.84.3:42609)
I20260810 14:42:59.824059 2364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:36951
I20260810 14:42:59.828819 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:42:59.829635 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 2489
I20260810 14:42:59.855901 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 2724
Aug 10 14:42:59 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372979, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:42:59.879822 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 2947
I20260810 14:42:59.893261 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 2310
I20260810 14:42:59.911819 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:39955
--webserver_interface=127.15.84.62
--webserver_port=34789
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:39955
--ranger_config_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.84.20:43847/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.84.20:43847/kms with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:00.047484 3155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:00.047783 3155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:00.047842 3155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:00.051956 3155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:43:00.052018 3155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:00.052044 3155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:00.052066 3155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:43:00.052096 3155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:43:00.057658 3155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.84.20:43847/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:39955
--ranger_config_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:39955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=34789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:00.058714 3155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:00.059751 3155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:00.068660 3168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:00.068707 3166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:00.068653 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:00.068694 3155 server_base.cc:1034] running on GCE node
I20260810 14:43:00.069485 3155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:00.070618 3155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:00.071789 3155 hybrid_clock.cc:648] HybridClock initialized: now 1786372980071757 us; error 48 us; skew 500 ppm
Aug 10 14:43:00 dist-test-slave-rmhg krb5kdc[1592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372980, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:00.074596 3155 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:43:00.075553 3155 webserver.cc:460] Webserver started at http://127.15.84.62:34789/ using document root <none> and password file <none>
I20260810 14:43:00.076081 3155 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:00.076150 3155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 14:43:00 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372980, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:43:00.152889 3155 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:43:00.155681 3176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:00.157112 3155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:43:00.157297 3155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "dc7301a291964706b44b5f7683788274"
format_stamp: "Formatted at 2026-08-10 14:42:57 on dist-test-slave-rmhg"
server_key: "4d38683fd4e6896b32b23f377b5348ce"
server_key_iv: "b04b20ab0ed45c4844704f94f79fd13f"
server_key_version: "kuduclusterkey@0"
I20260810 14:43:00.157915 3155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:00.179570 3155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:00.182806 3155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:00.183017 3155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:00.190547 3155 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:39955
I20260810 14:43:00.190555 3256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:39955 every 8 connection(s)
I20260810 14:43:00.191737 3155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:43:00.193325 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 3155
I20260810 14:43:00.193850 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.84.20:43847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:42817
--local_ip_for_outbound_sockets=127.15.84.1
--tserver_master_addrs=127.15.84.62:39955
--webserver_port=39739
--webserver_interface=127.15.84.1
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.84.20:43847/kms with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:43:00.197013 3257 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:43:00.198829 3257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274: Bootstrap starting.
I20260810 14:43:00.215969 3257 log.cc:824] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:00.223052 3257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:43:00.223460 3257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274: Bootstrap complete.
I20260810 14:43:00.227763 3257 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7301a291964706b44b5f7683788274" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39955 } }
I20260810 14:43:00.228197 3257 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7301a291964706b44b5f7683788274, State: Initialized, Role: FOLLOWER
I20260810 14:43:00.228749 3257 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7301a291964706b44b5f7683788274" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39955 } }
I20260810 14:43:00.228883 3257 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:43:00.228950 3257 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:43:00.229089 3257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:43:00.231479 3257 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7301a291964706b44b5f7683788274" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39955 } }
I20260810 14:43:00.231868 3257 leader_election.cc:302] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc7301a291964706b44b5f7683788274; no voters:
I20260810 14:43:00.232201 3257 leader_election.cc:288] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 14:43:00.232357 3266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:43:00.232627 3266 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [term 2 LEADER]: Becoming Leader. State: Replica: dc7301a291964706b44b5f7683788274, State: Running, Role: LEADER
I20260810 14:43:00.233067 3266 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7301a291964706b44b5f7683788274" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39955 } }
I20260810 14:43:00.233530 3257 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:43:00.234846 3267 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dc7301a291964706b44b5f7683788274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7301a291964706b44b5f7683788274" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39955 } } }
I20260810 14:43:00.235033 3267 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:00.234848 3270 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc7301a291964706b44b5f7683788274. Latest consensus state: current_term: 2 leader_uuid: "dc7301a291964706b44b5f7683788274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7301a291964706b44b5f7683788274" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 39955 } } }
I20260810 14:43:00.235397 3270 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274 [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:00.235992 3257 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 14:43:00.236044 3285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:43:00.243016 3285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:43:00.244073 3285 catalog_manager.cc:1259] Loaded cluster ID: e2d7f2740d1d44a28a0d42d0696eb033
I20260810 14:43:00.244163 3285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:43:00.247162 3285 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:43:00.247978 3285 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274: Loaded TSK: 0
I20260810 14:43:00.249527 3285 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:43:00.332628 3260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:00.332901 3260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:00.332963 3260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:00.338007 3260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:00.338084 3260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:00.338168 3260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:43:00.343798 3260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.84.20:43847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:42817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=39739
--tserver_master_addrs=127.15.84.62:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:00.344937 3260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:00.346167 3260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:00.358454 3329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:00.358845 3260 server_base.cc:1034] running on GCE node
W20260810 14:43:00.358454 3326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:00.359163 3324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:00.359599 3260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:00.360311 3260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:00.361536 3260 hybrid_clock.cc:648] HybridClock initialized: now 1786372980361505 us; error 45 us; skew 500 ppm
Aug 10 14:43:00 dist-test-slave-rmhg krb5kdc[1592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372980, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:00.364524 3260 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:43:00.365875 3260 webserver.cc:460] Webserver started at http://127.15.84.1:39739/ using document root <none> and password file <none>
I20260810 14:43:00.366508 3260 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:00.366618 3260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 14:43:00 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372980, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:43:00.398509 3260 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 14:43:00.402079 3344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:00.403792 3260 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 14:43:00.403930 3260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "d47deef295c34e7193d914d13b120332"
format_stamp: "Formatted at 2026-08-10 14:42:58 on dist-test-slave-rmhg"
server_key: "faff7aa0f93f22b70823b9a84a46235c"
server_key_iv: "feddb8bc66d8932ac60eeef3364ece09"
server_key_version: "kuduclusterkey@0"
I20260810 14:43:00.404698 3260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:00.431725 3260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:00.436434 3260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:00.436640 3260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:00.437285 3260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:00.438198 3260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:00.438251 3260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:00.438305 3260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:00.438330 3260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:00.449277 3260 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:42817
I20260810 14:43:00.449448 3542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:42817 every 8 connection(s)
I20260810 14:43:00.450795 3260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:43:00.453462 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 3260
I20260810 14:43:00.453936 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.84.20:43847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:44103
--local_ip_for_outbound_sockets=127.15.84.2
--tserver_master_addrs=127.15.84.62:39955
--webserver_port=44761
--webserver_interface=127.15.84.2
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.84.20:43847/kms with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:43:00 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372980, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:00.479060 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:00.456739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:40875 (local address 127.15.84.62:39955)
0810 14:43:00.457309 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:00.457336 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:00.458578 (+ 1242us) server_negotiation.cc:407] Connection header received
0810 14:43:00.460181 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:00.460201 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:00.460745 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:00.461237 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:00.463669 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:00.465638 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:00.467567 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:00.468377 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:00.468556 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:00.471991 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:00.471999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:00.472020 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:00.472060 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:00.474664 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:00.475406 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:00.475414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:00.475420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:00.475594 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:00.476031 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:00.476034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:00.476036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:00.476577 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:00.476689 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:00.477547 (+ 858us) server_negotiation.cc:299] Negotiation successful
0810 14:43:00.477922 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":149,"threads_started":1}
I20260810 14:43:00.480767 3552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:00.457793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:39955 (local address 127.15.84.1:40875)
0810 14:43:00.458333 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:00.458394 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:00.459577 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:00.462145 (+ 2568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:00.462155 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:00.462621 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:00.463415 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:00.463439 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:00.465855 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:00.465859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:00.467381 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:00.467398 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:00.468896 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:00.468901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:00.469130 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:00.469958 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:00.469982 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:00.471709 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:00.474846 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:00.474853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:00.474872 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:00.475249 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:00.475709 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:00.475711 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:00.475713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:00.475894 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:00.476770 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:00.476777 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:00.477124 (+ 347us) client_negotiation.cc:769] Sending connection context
0810 14:43:00.479446 (+ 2322us) client_negotiation.cc:240] Negotiation successful
0810 14:43:00.479822 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":144,"threads_started":1}
I20260810 14:43:00.483902 3545 heartbeater.cc:347] Connected to a master server at 127.15.84.62:39955
I20260810 14:43:00.484184 3545 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:00.484759 3545 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
W20260810 14:43:00.487161 3545 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:39955 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 14:43:00.487295 3545 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:00.487483 3545 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
I20260810 14:43:00.488521 3545 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:00.488718 3545 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
W20260810 14:43:00.489599 3545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:43:00.610852 3547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:00.611384 3547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:00.611478 3547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:00.616379 3547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:00.616456 3547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:00.616541 3547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:43:00.623517 3547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.84.20:43847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:44103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=44761
--tserver_master_addrs=127.15.84.62:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:00.625087 3547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:00.626562 3547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:00.639540 3565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:00.639566 3558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:00.639544 3562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:00.640588 3547 server_base.cc:1034] running on GCE node
I20260810 14:43:00.641206 3547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:00.642167 3547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:00.643368 3547 hybrid_clock.cc:648] HybridClock initialized: now 1786372980643329 us; error 52 us; skew 500 ppm
Aug 10 14:43:00 dist-test-slave-rmhg krb5kdc[1592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372980, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:00.647101 3547 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:43:00.648353 3547 webserver.cc:460] Webserver started at http://127.15.84.2:44761/ using document root <none> and password file <none>
I20260810 14:43:00.649212 3547 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:00.649333 3547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 14:43:00 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372980, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:43:00.673007 3547 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:43:00.676551 3580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:00.678498 3547 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 14:43:00.678658 3547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "68c85b5b33dd45cfb629d8ced69adb48"
format_stamp: "Formatted at 2026-08-10 14:42:59 on dist-test-slave-rmhg"
server_key: "96f3f37917402f4a675fbe4b41166bb1"
server_key_iv: "daa4ff280bf439621cc92fbd01d8b127"
server_key_version: "kuduclusterkey@0"
I20260810 14:43:00.679478 3547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:00.701262 3547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:00.705401 3547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:00.705687 3547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:00.706537 3547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:00.707746 3547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:00.707875 3547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:00.707993 3547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:00.708082 3547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:00.720291 3547 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:44103
I20260810 14:43:00.720562 3762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:44103 every 8 connection(s)
I20260810 14:43:00.721915 3547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:43:00.723486 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 3547
I20260810 14:43:00.724043 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.84.20:43847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:42609
--local_ip_for_outbound_sockets=127.15.84.3
--tserver_master_addrs=127.15.84.62:39955
--webserver_port=34979
--webserver_interface=127.15.84.3
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.84.20:43847/kms with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:43:00 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372980, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:00.746374 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:00.724264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:43681 (local address 127.15.84.62:39955)
0810 14:43:00.724415 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:00.724420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:00.726505 (+ 2085us) server_negotiation.cc:407] Connection header received
0810 14:43:00.727796 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:00.727800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:00.727862 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:00.727957 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:00.730975 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:00.731824 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:00.733773 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:00.734144 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:00.734264 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:00.737992 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:00.737997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:00.738013 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:00.738050 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:00.739834 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:00.740868 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:00.740872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:00.740874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:00.740978 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:00.741474 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:00.741478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:00.741480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:00.741736 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:00.741823 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:00.745961 (+ 4138us) server_negotiation.cc:299] Negotiation successful
0810 14:43:00.746177 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:43:00.748621 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:00.724846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:39955 (local address 127.15.84.2:43681)
0810 14:43:00.726328 (+ 1482us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:00.726389 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:00.727504 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:00.729540 (+ 2036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:00.729553 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:00.729949 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:00.730732 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:00.730759 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:00.732031 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:00.732035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:00.733600 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:00.733614 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:00.734518 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:00.734523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:00.734745 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:00.735489 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:00.735516 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:00.737444 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:00.740278 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:00.740286 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:00.740309 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:00.740734 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:00.741111 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:00.741114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:00.741116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:00.741356 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:00.741899 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:00.741906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:00.745481 (+ 3575us) client_negotiation.cc:769] Sending connection context
0810 14:43:00.746875 (+ 1394us) client_negotiation.cc:240] Negotiation successful
0810 14:43:00.747361 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"thread_start_us":136,"threads_started":1}
I20260810 14:43:00.750769 3763 heartbeater.cc:347] Connected to a master server at 127.15.84.62:39955
I20260810 14:43:00.751065 3763 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:00.751745 3763 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
W20260810 14:43:00.753964 3763 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:39955 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 14:43:00.754144 3763 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:00.754380 3763 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
I20260810 14:43:00.755613 3763 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:00.755838 3763 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
W20260810 14:43:00.756865 3763 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:43:00.878253 3768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:00.878552 3768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:00.878604 3768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:00.882964 3768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:00.883044 3768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:00.883109 3768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:43:00.889140 3768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.84.20:43847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:42609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=34979
--tserver_master_addrs=127.15.84.62:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:00.890434 3768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:00.891536 3768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:00.903165 3778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:00.903282 3780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:00.903165 3783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:00.903931 3768 server_base.cc:1034] running on GCE node
I20260810 14:43:00.904513 3768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:00.905406 3768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:00.906612 3768 hybrid_clock.cc:648] HybridClock initialized: now 1786372980906576 us; error 55 us; skew 500 ppm
Aug 10 14:43:00 dist-test-slave-rmhg krb5kdc[1592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372980, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:00.910620 3768 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:43:00.912035 3768 webserver.cc:460] Webserver started at http://127.15.84.3:34979/ using document root <none> and password file <none>
I20260810 14:43:00.912919 3768 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:00.913095 3768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 14:43:00 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372980, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:43:00.938199 3768 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:43:00.941133 3793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:00.942656 3768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:43:00.942799 3768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "3ff3156de3344ed0a98795960e3230f6"
format_stamp: "Formatted at 2026-08-10 14:42:59 on dist-test-slave-rmhg"
server_key: "281a5b1d15427cd994bd9b3f02b379f3"
server_key_iv: "d21a8679d73313d4913910f7fe933ab6"
server_key_version: "kuduclusterkey@0"
I20260810 14:43:00.943437 3768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:00.954798 3768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:00.959821 3768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:00.960120 3768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:00.960949 3768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:00.962479 3768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:00.962538 3768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:00.962699 3768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:00.962734 3768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:00.973904 3768 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:42609
I20260810 14:43:00.975206 3768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:43:00.976217 3974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:42609 every 8 connection(s)
I20260810 14:43:00.978479 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 3768
I20260810 14:43:00.991544 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:00.980895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.15.84.62:39955)
0810 14:43:00.981365 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:00.981369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:00.981381 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:43:00.981417 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:00.981419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:00.981460 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:00.981585 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:00.982590 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:00.983412 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:00.985059 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:00.985524 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:00.985636 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:00.987526 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:00.987530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:00.987532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:00.987562 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:00.989315 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:00.989924 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:00.989928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:00.989930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:00.990032 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:00.990351 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:00.990355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:00.990357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:00.990603 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:00.990827 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:00.991015 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 14:43:00.991179 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":109,"threads_started":1}
Aug 10 14:43:00 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372980, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:00.998108 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:00.980187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:48377 (local address 127.15.84.62:39955)
0810 14:43:00.980341 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:00.980345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:00.980367 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:43:00.980949 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:00.980954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:00.981014 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:00.981074 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:00.984243 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:00.984935 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:00.988878 (+ 3943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:00.989239 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:00.989330 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:00.994050 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:00.994054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:00.994057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:00.994087 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:00.995617 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:00.996284 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:00.996288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:00.996291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:00.996377 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:00.996809 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:00.996812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:00.996814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:00.997031 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:00.997123 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:00.997798 (+ 675us) server_negotiation.cc:299] Negotiation successful
0810 14:43:00.997940 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:43:00.999039 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:00.978978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:39955 (local address 127.15.84.3:48377)
0810 14:43:00.979540 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:00.979595 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:00.980665 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:00.981600 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:00.981610 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:00.981934 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:00.982510 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:00.984089 (+ 1579us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:00.985100 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:00.985104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:00.988710 (+ 3606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:00.988728 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:00.990041 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:00.990045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:00.990609 (+ 564us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:00.991994 (+ 1385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:00.992016 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:00.993838 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:00.995754 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:00.995763 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:00.995785 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:00.996156 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:00.996469 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:00.996473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:00.996475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:00.996675 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:00.997144 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:00.997178 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:00.997521 (+ 343us) client_negotiation.cc:769] Sending connection context
0810 14:43:00.997710 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 14:43:00.998105 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":141,"threads_started":1}
I20260810 14:43:01.000329 3975 heartbeater.cc:347] Connected to a master server at 127.15.84.62:39955
I20260810 14:43:01.000619 3975 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:01.001362 3975 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
W20260810 14:43:01.003206 3975 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:39955 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 14:43:01.003345 3975 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:01.003537 3975 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
I20260810 14:43:01.004686 3975 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:01.004899 3975 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
W20260810 14:43:01.005861 3975 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 14:43:01 dist-test-slave-rmhg krb5kdc[1592](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372981, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:01.489835 3545 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:01.490176 3545 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
I20260810 14:43:01.491396 3198 ts_manager.cc:194] Registered new tserver with Master: d47deef295c34e7193d914d13b120332 (127.15.84.1:42817)
I20260810 14:43:01.492826 3198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:40875
I20260810 14:43:01.757234 3763 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:01.757577 3763 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
I20260810 14:43:01.758635 3198 ts_manager.cc:194] Registered new tserver with Master: 68c85b5b33dd45cfb629d8ced69adb48 (127.15.84.2:44103)
I20260810 14:43:01.759471 3198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:43681
Aug 10 14:43:01 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372981, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
I20260810 14:43:02.006098 3975 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:02.006429 3975 heartbeater.cc:511] Master 127.15.84.62:39955 requested a full tablet report, sending...
I20260810 14:43:02.007292 3198 ts_manager.cc:194] Registered new tserver with Master: 3ff3156de3344ed0a98795960e3230f6 (127.15.84.3:42609)
I20260810 14:43:02.008097 3198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:48377
I20260810 14:43:02.019361 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:43:02.030463 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:02.021140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58336 (local address 127.15.84.62:39955)
0810 14:43:02.021431 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:02.021436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:02.021557 (+ 121us) server_negotiation.cc:407] Connection header received
0810 14:43:02.021695 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:02.021698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:02.021749 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:02.021838 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:02.022692 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.023496 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.024717 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.025064 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.025198 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:02.026837 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:02.026842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:02.026845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:02.026876 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:02.028603 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:02.029105 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:02.029109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:02.029111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:02.029231 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:02.029566 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:02.029569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:02.029571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:02.029794 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:02.029918 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:02.030153 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 14:43:02.030284 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
I20260810 14:43:02.036747 3198 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:43:02.041712 3198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:43:02.065968 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:02.056622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38030 (local address 127.15.84.3:42609)
0810 14:43:02.058874 (+ 2252us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:02.058881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:02.059225 (+ 344us) server_negotiation.cc:407] Connection header received
0810 14:43:02.059324 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:02.059330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:02.060003 (+ 673us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:02.060169 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:02.060733 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.062122 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.064011 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.064918 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.065006 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:02.065199 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:02.065683 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 14:43:02.065797 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2148,"thread_start_us":97,"threads_started":1}
I20260810 14:43:02.066519 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:02.056449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:42609 (local address 127.0.0.1:38030)
0810 14:43:02.059114 (+ 2665us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:02.059126 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:02.059210 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:02.060383 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:02.060389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:02.060415 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:02.060563 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:02.060572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.062278 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.062281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:02.063852 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:02.063865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.065304 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.065307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:02.065429 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:02.065477 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 14:43:02.066240 (+ 763us) client_negotiation.cc:240] Negotiation successful
0810 14:43:02.066381 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2606,"thread_start_us":101,"threads_started":1}
I20260810 14:43:02.068738 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:02.056375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:44103 (local address 127.0.0.1:55578)
0810 14:43:02.058174 (+ 1799us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:02.058228 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:02.058357 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:02.061908 (+ 3551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:02.061911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:02.061930 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:02.062029 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:02.062035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.064600 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.064604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:02.065795 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:02.065807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.068345 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.068349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:02.068437 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:02.068461 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:43:02.068538 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:43:02.068605 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1606,"thread_start_us":116,"threads_started":1}
I20260810 14:43:02.069352 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:02.057334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.15.84.2:44103)
0810 14:43:02.061408 (+ 4074us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:02.061415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:02.061440 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:43:02.061515 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:02.061520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:02.061752 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:02.062272 (+ 520us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:02.062490 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.064435 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.065953 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.068219 (+ 2266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.068326 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:02.068461 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:02.069080 (+ 619us) server_negotiation.cc:299] Negotiation successful
0810 14:43:02.069220 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3953,"thread_start_us":96,"threads_started":1}
I20260810 14:43:02.070125 3845 tablet_service.cc:1467] Processing CreateTablet for tablet c11ef724de614709b9c89d31cbcd686d (DEFAULT_TABLE table=test-table [id=edd61cc577bc4c4db347699b5a174b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:02.071352 3845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11ef724de614709b9c89d31cbcd686d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:02.072332 3643 tablet_service.cc:1467] Processing CreateTablet for tablet c11ef724de614709b9c89d31cbcd686d (DEFAULT_TABLE table=test-table [id=edd61cc577bc4c4db347699b5a174b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:02.073386 3643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11ef724de614709b9c89d31cbcd686d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:02.083876 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:02.058492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38556 (local address 127.15.84.1:42817)
0810 14:43:02.058781 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:02.058788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:02.066622 (+ 7834us) server_negotiation.cc:407] Connection header received
0810 14:43:02.066723 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:02.066728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:02.066943 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:02.067098 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:02.067509 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.070305 (+ 2796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.082291 (+ 11986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.082968 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.083045 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:02.083197 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:02.083652 (+ 455us) server_negotiation.cc:299] Negotiation successful
0810 14:43:02.083753 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260810 14:43:02.084327 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:02.058087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:42817 (local address 127.0.0.1:38556)
0810 14:43:02.066375 (+ 8288us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:02.066390 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:02.066492 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:02.067286 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:02.067288 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:02.067304 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:02.067386 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:02.067392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.077397 (+ 10005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.077402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:02.078563 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:02.078577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.083386 (+ 4809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.083388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:02.083476 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:02.083500 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:43:02.084094 (+ 594us) client_negotiation.cc:240] Negotiation successful
0810 14:43:02.084196 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8223,"thread_start_us":97,"threads_started":1}
I20260810 14:43:02.087620 3415 tablet_service.cc:1467] Processing CreateTablet for tablet c11ef724de614709b9c89d31cbcd686d (DEFAULT_TABLE table=test-table [id=edd61cc577bc4c4db347699b5a174b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:02.088806 3415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11ef724de614709b9c89d31cbcd686d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:02.089775 4018 tablet_bootstrap.cc:492] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6: Bootstrap starting.
I20260810 14:43:02.092192 4018 tablet_bootstrap.cc:654] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:02.093339 4016 tablet_bootstrap.cc:492] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48: Bootstrap starting.
I20260810 14:43:02.095554 4016 tablet_bootstrap.cc:654] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:02.096428 4016 log.cc:824] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:02.097282 4018 log.cc:824] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:02.100704 4016 tablet_bootstrap.cc:492] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48: No bootstrap required, opened a new log
I20260810 14:43:02.101104 4020 tablet_bootstrap.cc:492] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332: Bootstrap starting.
I20260810 14:43:02.103497 4020 tablet_bootstrap.cc:654] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:02.104285 4016 ts_tablet_manager.cc:1397] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260810 14:43:02.106029 4018 tablet_bootstrap.cc:492] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6: No bootstrap required, opened a new log
I20260810 14:43:02.106293 4018 ts_tablet_manager.cc:1397] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260810 14:43:02.107121 4020 log.cc:824] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:02.109863 4016 raft_consensus.cc:348] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c85b5b33dd45cfb629d8ced69adb48" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44103 } } peers { permanent_uuid: "3ff3156de3344ed0a98795960e3230f6" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42609 } } peers { permanent_uuid: "d47deef295c34e7193d914d13b120332" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42817 } }
I20260810 14:43:02.110286 4018 raft_consensus.cc:348] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c85b5b33dd45cfb629d8ced69adb48" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44103 } } peers { permanent_uuid: "3ff3156de3344ed0a98795960e3230f6" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42609 } } peers { permanent_uuid: "d47deef295c34e7193d914d13b120332" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42817 } }
I20260810 14:43:02.110646 4018 raft_consensus.cc:374] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:02.110746 4018 raft_consensus.cc:729] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff3156de3344ed0a98795960e3230f6, State: Initialized, Role: FOLLOWER
I20260810 14:43:02.110913 4016 raft_consensus.cc:374] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:02.111028 4016 raft_consensus.cc:729] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c85b5b33dd45cfb629d8ced69adb48, State: Initialized, Role: FOLLOWER
I20260810 14:43:02.111395 4018 consensus_queue.cc:260] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c85b5b33dd45cfb629d8ced69adb48" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44103 } } peers { permanent_uuid: "3ff3156de3344ed0a98795960e3230f6" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42609 } } peers { permanent_uuid: "d47deef295c34e7193d914d13b120332" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42817 } }
I20260810 14:43:02.111703 4016 consensus_queue.cc:260] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c85b5b33dd45cfb629d8ced69adb48" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44103 } } peers { permanent_uuid: "3ff3156de3344ed0a98795960e3230f6" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42609 } } peers { permanent_uuid: "d47deef295c34e7193d914d13b120332" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42817 } }
I20260810 14:43:02.112619 4016 ts_tablet_manager.cc:1428] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260810 14:43:02.112619 4018 ts_tablet_manager.cc:1428] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:43:02.113027 3763 heartbeater.cc:503] Master 127.15.84.62:39955 was elected leader, sending a full tablet report...
I20260810 14:43:02.113029 3975 heartbeater.cc:503] Master 127.15.84.62:39955 was elected leader, sending a full tablet report...
I20260810 14:43:02.120640 4020 tablet_bootstrap.cc:492] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332: No bootstrap required, opened a new log
I20260810 14:43:02.120896 4020 ts_tablet_manager.cc:1397] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20260810 14:43:02.125191 4020 raft_consensus.cc:348] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c85b5b33dd45cfb629d8ced69adb48" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44103 } } peers { permanent_uuid: "3ff3156de3344ed0a98795960e3230f6" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42609 } } peers { permanent_uuid: "d47deef295c34e7193d914d13b120332" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42817 } }
I20260810 14:43:02.125581 4020 raft_consensus.cc:374] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:02.125658 4020 raft_consensus.cc:729] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47deef295c34e7193d914d13b120332, State: Initialized, Role: FOLLOWER
I20260810 14:43:02.126977 4020 consensus_queue.cc:260] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c85b5b33dd45cfb629d8ced69adb48" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44103 } } peers { permanent_uuid: "3ff3156de3344ed0a98795960e3230f6" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42609 } } peers { permanent_uuid: "d47deef295c34e7193d914d13b120332" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42817 } }
I20260810 14:43:02.127820 3545 heartbeater.cc:503] Master 127.15.84.62:39955 was elected leader, sending a full tablet report...
I20260810 14:43:02.128234 4020 ts_tablet_manager.cc:1428] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
Aug 10 14:43:02 dist-test-slave-rmhg krb5kdc[1592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372981, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for HTTP/127.15.84.20@KRBTEST.COM
W20260810 14:43:02.223393 3546 tablet.cc:2369] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:43:02.227993 3976 tablet.cc:2369] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:43:02.230064 3765 tablet.cc:2369] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:43:02.449463 4025 raft_consensus.cc:482] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:43:02.449735 4025 raft_consensus.cc:504] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c85b5b33dd45cfb629d8ced69adb48" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44103 } } peers { permanent_uuid: "3ff3156de3344ed0a98795960e3230f6" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42609 } } peers { permanent_uuid: "d47deef295c34e7193d914d13b120332" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42817 } }
I20260810 14:43:02.451282 4025 leader_election.cc:288] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ff3156de3344ed0a98795960e3230f6 (127.15.84.3:42609), d47deef295c34e7193d914d13b120332 (127.15.84.1:42817)
I20260810 14:43:02.456825 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:02.451909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:57657 (local address 127.15.84.3:42609)
0810 14:43:02.452038 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:02.452042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:02.452380 (+ 338us) server_negotiation.cc:407] Connection header received
0810 14:43:02.452570 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:02.452574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:02.452624 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:02.452672 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:02.453093 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.453941 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.455536 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.456154 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.456230 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:02.456300 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:02.456615 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 14:43:02.456693 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:43:02.457283 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:02.451511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:42609 (local address 127.15.84.2:57657)
0810 14:43:02.452293 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:02.452307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:02.452456 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:02.452833 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:02.452836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:02.452853 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:02.452987 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:02.452992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.454080 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.454083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:02.455390 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:02.455401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.456389 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.456392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:02.456488 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:02.456511 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:43:02.457031 (+ 520us) client_negotiation.cc:240] Negotiation successful
0810 14:43:02.457111 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":132,"threads_started":1}
I20260810 14:43:02.457302 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:02.452428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:37881 (local address 127.15.84.1:42817)
0810 14:43:02.452564 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:02.452569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:02.452858 (+ 289us) server_negotiation.cc:407] Connection header received
0810 14:43:02.453052 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:02.453056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:02.453102 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:02.453191 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:02.453784 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.454533 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.455945 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.456549 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.456653 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:02.456718 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:02.457065 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 14:43:02.457180 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:43:02.457724 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:02.452057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:42817 (local address 127.15.84.2:37881)
0810 14:43:02.452761 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:02.452777 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:02.452935 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:02.453505 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:02.453508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:02.453527 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:02.453654 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:02.453661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.454675 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.454683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:02.455807 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:02.455819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.456820 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.456823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:02.456925 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:02.456949 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:43:02.457490 (+ 541us) client_negotiation.cc:240] Negotiation successful
0810 14:43:02.457563 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":131,"threads_started":1}
I20260810 14:43:02.457945 3873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11ef724de614709b9c89d31cbcd686d" candidate_uuid: "68c85b5b33dd45cfb629d8ced69adb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff3156de3344ed0a98795960e3230f6" is_pre_election: true
I20260810 14:43:02.458294 3873 raft_consensus.cc:2454] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68c85b5b33dd45cfb629d8ced69adb48 in term 0.
I20260810 14:43:02.458678 3457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11ef724de614709b9c89d31cbcd686d" candidate_uuid: "68c85b5b33dd45cfb629d8ced69adb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d47deef295c34e7193d914d13b120332" is_pre_election: true
I20260810 14:43:02.458750 3588 leader_election.cc:302] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ff3156de3344ed0a98795960e3230f6, 68c85b5b33dd45cfb629d8ced69adb48; no voters:
I20260810 14:43:02.458981 3457 raft_consensus.cc:2454] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68c85b5b33dd45cfb629d8ced69adb48 in term 0.
I20260810 14:43:02.459041 4025 raft_consensus.cc:2781] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:43:02.459136 4025 raft_consensus.cc:482] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:43:02.459187 4025 raft_consensus.cc:3037] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:02.460281 4025 raft_consensus.cc:504] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c85b5b33dd45cfb629d8ced69adb48" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44103 } } peers { permanent_uuid: "3ff3156de3344ed0a98795960e3230f6" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42609 } } peers { permanent_uuid: "d47deef295c34e7193d914d13b120332" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42817 } }
I20260810 14:43:02.460628 4025 leader_election.cc:288] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [CANDIDATE]: Term 1 election: Requested vote from peers 3ff3156de3344ed0a98795960e3230f6 (127.15.84.3:42609), d47deef295c34e7193d914d13b120332 (127.15.84.1:42817)
I20260810 14:43:02.460966 3873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11ef724de614709b9c89d31cbcd686d" candidate_uuid: "68c85b5b33dd45cfb629d8ced69adb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff3156de3344ed0a98795960e3230f6"
I20260810 14:43:02.460984 3457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11ef724de614709b9c89d31cbcd686d" candidate_uuid: "68c85b5b33dd45cfb629d8ced69adb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d47deef295c34e7193d914d13b120332"
I20260810 14:43:02.461089 3873 raft_consensus.cc:3037] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:02.461088 3457 raft_consensus.cc:3037] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:02.462364 3457 raft_consensus.cc:2454] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68c85b5b33dd45cfb629d8ced69adb48 in term 1.
I20260810 14:43:02.462364 3873 raft_consensus.cc:2454] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68c85b5b33dd45cfb629d8ced69adb48 in term 1.
I20260810 14:43:02.462653 3582 leader_election.cc:302] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68c85b5b33dd45cfb629d8ced69adb48, d47deef295c34e7193d914d13b120332; no voters:
I20260810 14:43:02.462819 4025 raft_consensus.cc:2781] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:43:02.463003 4025 raft_consensus.cc:686] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [term 1 LEADER]: Becoming Leader. State: Replica: 68c85b5b33dd45cfb629d8ced69adb48, State: Running, Role: LEADER
I20260810 14:43:02.463335 4025 consensus_queue.cc:237] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c85b5b33dd45cfb629d8ced69adb48" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44103 } } peers { permanent_uuid: "3ff3156de3344ed0a98795960e3230f6" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42609 } } peers { permanent_uuid: "d47deef295c34e7193d914d13b120332" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42817 } }
I20260810 14:43:02.466807 3197 catalog_manager.cc:5495] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68c85b5b33dd45cfb629d8ced69adb48 (127.15.84.2). New cstate: current_term: 1 leader_uuid: "68c85b5b33dd45cfb629d8ced69adb48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68c85b5b33dd45cfb629d8ced69adb48" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ff3156de3344ed0a98795960e3230f6" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d47deef295c34e7193d914d13b120332" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42817 } health_report { overall_health: UNKNOWN } } }
I20260810 14:43:02.517947 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:02.514610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.15.84.2:44103)
0810 14:43:02.514740 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:02.514744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:02.514757 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:43:02.514881 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:02.514883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:02.514944 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:02.515037 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:43:02.515232 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.515741 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.516539 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:02.516768 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:02.516841 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:02.517088 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:43:02.517565 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:43:02.517667 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:02.517718 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 14:43:02.517777 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:43:02.524202 3457 raft_consensus.cc:1261] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332 [term 1 FOLLOWER]: Refusing update from remote peer 68c85b5b33dd45cfb629d8ced69adb48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:43:02.524202 3873 raft_consensus.cc:1261] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6 [term 1 FOLLOWER]: Refusing update from remote peer 68c85b5b33dd45cfb629d8ced69adb48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:43:02.525012 4047 consensus_queue.cc:1035] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d47deef295c34e7193d914d13b120332" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:43:02.525235 4025 consensus_queue.cc:1035] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff3156de3344ed0a98795960e3230f6" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:43:02.532063 4051 mvcc.cc:205] Tried to move back new op lower bound from 7316983736411209728 to 7316983736172445696. Current Snapshot: MvccSnapshot[applied={T|T < 7316983736411209728}]
I20260810 14:43:02.535383 4052 mvcc.cc:205] Tried to move back new op lower bound from 7316983736411209728 to 7316983736172445696. Current Snapshot: MvccSnapshot[applied={T|T < 7316983736411209728}]
I20260810 14:43:02.541703 3197 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:43:02.541886 3197 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:43:02.544413 3197 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc7301a291964706b44b5f7683788274: Sending DeleteTablet for 3 replicas of tablet c11ef724de614709b9c89d31cbcd686d
I20260810 14:43:02.545346 3643 tablet_service.cc:1514] Processing DeleteTablet for tablet c11ef724de614709b9c89d31cbcd686d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:02 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:55578
I20260810 14:43:02.545315 3845 tablet_service.cc:1514] Processing DeleteTablet for tablet c11ef724de614709b9c89d31cbcd686d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:02 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:38030
I20260810 14:43:02.545311 3415 tablet_service.cc:1514] Processing DeleteTablet for tablet c11ef724de614709b9c89d31cbcd686d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:02 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:38556
I20260810 14:43:02.545976 4066 tablet_replica.cc:330] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332: stopping tablet replica
I20260810 14:43:02.546113 4069 tablet_replica.cc:330] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48: stopping tablet replica
I20260810 14:43:02.546393 4066 raft_consensus.cc:2229] T c11ef724de614709b9c89d31cbcd686d P d47deef295c34e7193d914d13b120332 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:43:02.546437 4069 raft_consensus.cc:2229] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:43:02.546718 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 3260
I20260810 14:43:02.546825 4069 raft_consensus.cc:2258] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:43:02.547226 4068 tablet_replica.cc:330] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6: stopping tablet replica
I20260810 14:43:02.547545 4068 raft_consensus.cc:2229] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:43:02.547834 4068 raft_consensus.cc:2258] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:43:02.577481 4069 ts_tablet_manager.cc:1905] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:43:02.580528 4069 ts_tablet_manager.cc:1918] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:43:02.580659 4069 log.cc:1196] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/c11ef724de614709b9c89d31cbcd686d
I20260810 14:43:02.580986 4069 ts_tablet_manager.cc:1939] T c11ef724de614709b9c89d31cbcd686d P 68c85b5b33dd45cfb629d8ced69adb48: Deleting consensus metadata
I20260810 14:43:02.584800 3180 catalog_manager.cc:4841] TS 68c85b5b33dd45cfb629d8ced69adb48 (127.15.84.2:44103): tablet c11ef724de614709b9c89d31cbcd686d (table test-table [id=edd61cc577bc4c4db347699b5a174b7f]) successfully deleted
I20260810 14:43:02.587766 4068 ts_tablet_manager.cc:1905] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:43:02.590791 4068 ts_tablet_manager.cc:1918] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:43:02.590917 4068 log.cc:1196] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/c11ef724de614709b9c89d31cbcd686d
I20260810 14:43:02.591288 4068 ts_tablet_manager.cc:1939] T c11ef724de614709b9c89d31cbcd686d P 3ff3156de3344ed0a98795960e3230f6: Deleting consensus metadata
W20260810 14:43:02.592415 3178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:42817 (error 108)
I20260810 14:43:02.592492 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 3547
I20260810 14:43:02.597728 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:02.597254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:42817 (local address 127.0.0.1:38566)
0810 14:43:02.597503 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:02.597591 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
I20260810 14:43:02.597684 3181 catalog_manager.cc:4841] TS 3ff3156de3344ed0a98795960e3230f6 (127.15.84.3:42609): tablet c11ef724de614709b9c89d31cbcd686d (table test-table [id=edd61cc577bc4c4db347699b5a174b7f]) successfully deleted
W20260810 14:43:02.599200 3178 catalog_manager.cc:4584] TS d47deef295c34e7193d914d13b120332 (127.15.84.1:42817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c11ef724de614709b9c89d31cbcd686d: Network error: Client connection negotiation failed: client connection to 127.15.84.1:42817: connect: Connection refused (error 111)
I20260810 14:43:02.604872 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 3768
I20260810 14:43:02.615617 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 3155
I20260810 14:43:02.634325 15696 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 14:43:02.982823 15696 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 14:43:02.982949 15696 mini_ranger.cc:66] Stopping Ranger...
I20260810 14:43:03.328373 15696 mini_ranger.cc:68] Stopped Ranger
2026-08-10 14:43:03.328 UTC [1644] LOG: received smart shutdown request
2026-08-10 14:43:03.333 UTC [1644] LOG: background worker "logical replication launcher" (PID 1659) exited with exit code 1
2026-08-10 14:43:03.333 UTC [1650] LOG: shutting down
2026-08-10 14:43:03.494 UTC [1644] LOG: database system is shut down
2026-08-10T14:43:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50630 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:43:03 dist-test-slave-rmhg krb5kdc[4090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:43:03 dist-test-slave-rmhg krb5kdc[4090](info): set up 2 sockets
Aug 10 14:43:03 dist-test-slave-rmhg krb5kdc[4090](info): commencing operation
krb5kdc: starting...
W20260810 14:43:04.748461 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:43:04 dist-test-slave-rmhg krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:43:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:43:04.880374 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:44795
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:44795,127.15.84.61:40803,127.15.84.60:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:04.996721 4112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:04.997011 4112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:04.997081 4112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:05.001179 4112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:43:05.001248 4112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:05.001276 4112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:05.001298 4112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:43:05.001327 4112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:43:05.006763 4112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:44795,127.15.84.61:40803,127.15.84.60:43783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:44795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:05.007875 4112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:05.008929 4112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:05.017603 4121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:05.017828 4112 server_base.cc:1034] running on GCE node
W20260810 14:43:05.017598 4117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:05.017616 4118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:05.018630 4112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:05.019812 4112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:05.020974 4112 hybrid_clock.cc:648] HybridClock initialized: now 1786372985020943 us; error 54 us; skew 500 ppm
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372985, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:05.023744 4112 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:43:05.024683 4112 webserver.cc:460] Webserver started at http://127.15.84.62:46085/ using document root <none> and password file <none>
I20260810 14:43:05.025285 4112 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:05.025362 4112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:05.025574 4112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:43:05.028113 4112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "79192ecdfdf745b6922f4ffd16d6bc89"
format_stamp: "Formatted at 2026-08-10 14:43:05 on dist-test-slave-rmhg"
server_key: "910de06ea63392f069096aeed0689b70"
server_key_iv: "2b61984056dd382d9daeeddcbf6f30ff"
server_key_version: "encryptionkey@0"
I20260810 14:43:05.028684 4112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "79192ecdfdf745b6922f4ffd16d6bc89"
format_stamp: "Formatted at 2026-08-10 14:43:05 on dist-test-slave-rmhg"
server_key: "910de06ea63392f069096aeed0689b70"
server_key_iv: "2b61984056dd382d9daeeddcbf6f30ff"
server_key_version: "encryptionkey@0"
I20260810 14:43:05.032696 4112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 14:43:05.035260 4131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:05.036458 4112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:43:05.036577 4112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "79192ecdfdf745b6922f4ffd16d6bc89"
format_stamp: "Formatted at 2026-08-10 14:43:05 on dist-test-slave-rmhg"
server_key: "910de06ea63392f069096aeed0689b70"
server_key_iv: "2b61984056dd382d9daeeddcbf6f30ff"
server_key_version: "encryptionkey@0"
I20260810 14:43:05.036674 4112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:05.077240 4112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:05.080217 4112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:05.080415 4112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:05.087801 4112 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:44795
I20260810 14:43:05.087854 4213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:44795 every 8 connection(s)
I20260810 14:43:05.088835 4112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:43:05.091774 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:05.095767 4214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:43:05.098608 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 4112
I20260810 14:43:05.098760 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:43:05.099050 15696 external_mini_cluster.cc:1428] Setting key bb27ca448c19b8da432340c4fa42b15a
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372985, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.61@KRBTEST.COM" created.
I20260810 14:43:05.112108 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.097982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44795 (local address 127.0.0.1:49514)
0810 14:43:05.098481 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.098543 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:05.099608 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:05.100316 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:05.100321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:05.100773 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:05.101167 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.101191 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.102799 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.102802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.103560 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.103568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.104509 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.104512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.104595 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.105211 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:05.105231 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:05.106924 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:05.109010 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.109019 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.109028 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.109323 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.109694 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.109697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.109698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.109815 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.110437 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:05.110442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:05.110580 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 14:43:05.110858 (+ 278us) client_negotiation.cc:240] Negotiation successful
0810 14:43:05.111284 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":112,"thread_start_us":143,"threads_started":1}
I20260810 14:43:05.112129 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.098776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.15.84.62:44795)
0810 14:43:05.099047 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.099057 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.099094 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:43:05.099925 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.099930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.100120 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.100274 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.101348 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.102675 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.103692 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.104345 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.104544 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.107096 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.107104 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.107120 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.107148 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.108897 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.109450 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.109455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.109457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.109600 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.109896 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.109899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.109900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.110334 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.110444 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.110921 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.111152 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
I20260810 14:43:05.114990 4214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:43:05.115767 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.115467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:40803 (local address 127.0.0.1:42000)
0810 14:43:05.115602 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.115692 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:40803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 14:43:05.116010 4136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.84.61:40803: connect: Connection refused (error 111)
I20260810 14:43:05.116878 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.116669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:40803 (local address 127.0.0.1:42012)
0810 14:43:05.116778 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.116823 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:40803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 14:43:05.117007 4214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.61:40803: Network error: Client connection negotiation failed: client connection to 127.15.84.61:40803: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.61.keytab.
Entry for principal kudu/127.15.84.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.61.keytab.
I20260810 14:43:05.128691 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.61:40803
--webserver_interface=127.15.84.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.61.keytab
--principal=kudu/127.15.84.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:44795,127.15.84.61:40803,127.15.84.60:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:43:05.166286 4214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } attempt: 1
I20260810 14:43:05.167261 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.166889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:40803 (local address 127.0.0.1:42020)
0810 14:43:05.167070 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.167150 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:40803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 14:43:05.168088 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.167842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:40803 (local address 127.0.0.1:42024)
0810 14:43:05.167955 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.168013 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:40803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 14:43:05.168255 4214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.61:40803: Network error: Client connection negotiation failed: client connection to 127.15.84.61:40803: connect: Connection refused (error 111)
I20260810 14:43:05.236445 4214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } attempt: 2
I20260810 14:43:05.237397 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.237015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:40803 (local address 127.0.0.1:42034)
0810 14:43:05.237223 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.237298 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:40803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 14:43:05.238127 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.237891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:40803 (local address 127.0.0.1:42042)
0810 14:43:05.237997 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.238052 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:40803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 14:43:05.238265 4214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.61:40803: Network error: Client connection negotiation failed: client connection to 127.15.84.61:40803: connect: Connection refused (error 111)
W20260810 14:43:05.248230 4231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:05.248505 4231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:05.248546 4231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:05.252573 4231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:43:05.252640 4231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:05.252660 4231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:05.252676 4231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:43:05.252691 4231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:43:05.258121 4231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:44795,127.15.84.61:40803,127.15.84.60:43783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.61.keytab
--principal=kudu/127.15.84.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.61:40803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.84.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:05.259196 4231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:05.260207 4231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:05.268739 4244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:05.268893 4231 server_base.cc:1034] running on GCE node
W20260810 14:43:05.268751 4238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:05.268757 4241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:05.269630 4231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:05.270795 4231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:05.271946 4231 hybrid_clock.cc:648] HybridClock initialized: now 1786372985271924 us; error 48 us; skew 500 ppm
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372985, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:05.274757 4231 init.cc:377] Logged in from keytab as kudu/127.15.84.61@KRBTEST.COM (short username kudu)
I20260810 14:43:05.275734 4231 webserver.cc:460] Webserver started at http://127.15.84.61:44683/ using document root <none> and password file <none>
I20260810 14:43:05.276317 4231 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:05.276382 4231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:05.276593 4231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:43:05.278700 4231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/data/instance:
uuid: "6bddb60b720f419bbfb6bd1f23da003c"
format_stamp: "Formatted at 2026-08-10 14:43:05 on dist-test-slave-rmhg"
server_key: "fc8e6fa7753ea2030ae73bc110845f8b"
server_key_iv: "7950a1338831a8533878ffc4c77d9594"
server_key_version: "encryptionkey@0"
I20260810 14:43:05.279264 4231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/wal/instance:
uuid: "6bddb60b720f419bbfb6bd1f23da003c"
format_stamp: "Formatted at 2026-08-10 14:43:05 on dist-test-slave-rmhg"
server_key: "fc8e6fa7753ea2030ae73bc110845f8b"
server_key_iv: "7950a1338831a8533878ffc4c77d9594"
server_key_version: "encryptionkey@0"
I20260810 14:43:05.283043 4231 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 14:43:05.285482 4255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:05.286717 4231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:43:05.286832 4231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/wal
uuid: "6bddb60b720f419bbfb6bd1f23da003c"
format_stamp: "Formatted at 2026-08-10 14:43:05 on dist-test-slave-rmhg"
server_key: "fc8e6fa7753ea2030ae73bc110845f8b"
server_key_iv: "7950a1338831a8533878ffc4c77d9594"
server_key_version: "encryptionkey@0"
I20260810 14:43:05.286933 4231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:05.305454 4231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:05.308418 4231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:05.308602 4231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:05.316042 4231 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.61:40803
I20260810 14:43:05.316098 4343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.61:40803 every 8 connection(s)
I20260810 14:43:05.317123 4231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/data/info.pb
I20260810 14:43:05.319984 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:05.323807 4346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:43:05.326367 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 4231
I20260810 14:43:05.326512 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-1/wal/instance
I20260810 14:43:05.326840 15696 external_mini_cluster.cc:1428] Setting key d6a4458d5f14882920cd11eb3aae75a1
I20260810 14:43:05.329413 4214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } attempt: 3
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372985, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.61@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372985, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for kudu/127.15.84.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.60@KRBTEST.COM; defaulting to no policy
I20260810 14:43:05.339622 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.325719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.15.84.62:44795)
0810 14:43:05.325860 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.325863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.326767 (+ 904us) server_negotiation.cc:407] Connection header received
0810 14:43:05.327968 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.327971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.328027 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.328101 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.329702 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.330419 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.331777 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.332023 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.332111 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.335039 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.335042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.335044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.335068 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.336504 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.337127 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.337130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.337132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.337248 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.337618 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.337622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.337624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.337818 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.337907 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.339332 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.339461 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:43:05.339943 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.326040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44795 (local address 127.0.0.1:49520)
0810 14:43:05.326586 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.326648 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:05.327724 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:05.328311 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:05.328321 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:05.328779 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:05.329520 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.329544 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.330550 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.330553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.331664 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.331674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.332169 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.332173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.332405 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.333027 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:05.333050 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:05.334902 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:05.336620 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.336627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.336643 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.337017 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.337343 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.337347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.337349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.337527 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.337925 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:05.337933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:05.338291 (+ 358us) client_negotiation.cc:769] Sending connection context
0810 14:43:05.338525 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 14:43:05.338866 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.60@KRBTEST.COM" created.
I20260810 14:43:05.340581 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.329890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:40803 (local address 127.0.0.1:42046)
0810 14:43:05.330077 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.330092 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:05.330167 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:05.330894 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:05.330897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:05.331090 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:05.331216 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.331224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.332777 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.332780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.333401 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.333413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.334048 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.334053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.334171 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.334616 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:05.334632 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:05.335980 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:05.338843 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.338846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.338849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.339131 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.339445 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.339448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.339450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.339597 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.340119 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:05.340123 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:05.340223 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 14:43:05.340305 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 14:43:05.340438 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260810 14:43:05.341332 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.329996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.15.84.61:40803)
0810 14:43:05.330414 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.330426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.330450 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:43:05.330492 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.330497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.330721 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.330884 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.331342 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.332631 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.333519 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.333906 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.334019 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.336648 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.336657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.336671 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.336693 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.338733 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.339268 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.339274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.339276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.339362 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.339695 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.339698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.339700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.339990 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.340907 (+ 917us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.341033 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.341194 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":106,"threads_started":1}
I20260810 14:43:05.341665 4346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:43:05.343266 4214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:43:05.343922 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.343694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34786)
0810 14:43:05.343801 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.343855 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 14:43:05.344646 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.344359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34788)
0810 14:43:05.344531 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.344585 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260810 14:43:05.344849 4214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:43783: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372985, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.61@KRBTEST.COM for kudu/127.15.84.61@KRBTEST.COM
I20260810 14:43:05.350539 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.342166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:40803 (local address 127.0.0.1:42060)
0810 14:43:05.342283 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.342298 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:05.342375 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:05.342700 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:05.342703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:05.342864 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:05.342974 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.342981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.343976 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.343979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.344810 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.344821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.345449 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.345452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.345538 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.345969 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:05.345982 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:05.347384 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:05.349233 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.349235 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.349237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.349416 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.349693 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.349695 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.349696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.349762 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.350160 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:05.350162 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:05.350237 (+ 75us) client_negotiation.cc:769] Sending connection context
0810 14:43:05.350309 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 14:43:05.350410 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 14:43:05.350692 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.342339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.15.84.61:40803)
0810 14:43:05.342433 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.342438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.342450 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:43:05.342482 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.342485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.342528 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.342601 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.343100 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.343837 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.344960 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.345315 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.345420 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.347528 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.347533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.347536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.347563 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.349098 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.349510 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.349514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.349516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.349605 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.349843 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.349846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.349847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.350072 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.350175 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.350390 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.350514 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 14:43:05.351416 4346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:43:05.352046 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.351807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34790)
0810 14:43:05.351913 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.351990 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 14:43:05.352603 4260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
I20260810 14:43:05.353526 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.353337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34806)
0810 14:43:05.353435 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.353478 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 14:43:05.353654 4346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:43783: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.60.keytab.
Entry for principal kudu/127.15.84.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.60.keytab.
I20260810 14:43:05.359496 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.60:43783
--webserver_interface=127.15.84.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.60.keytab
--principal=kudu/127.15.84.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:44795,127.15.84.61:40803,127.15.84.60:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:43:05.376015 4214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } attempt: 1
I20260810 14:43:05.376965 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.376621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34810)
0810 14:43:05.376796 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.376869 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 14:43:05.377671 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.377457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34816)
0810 14:43:05.377549 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.377598 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 14:43:05.377791 4214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:43783: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
I20260810 14:43:05.402957 4346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } attempt: 1
I20260810 14:43:05.403844 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.403541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34820)
0810 14:43:05.403694 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.403761 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 14:43:05.404709 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.404482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34832)
0810 14:43:05.404599 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.404647 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 14:43:05.404870 4346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:43783: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
I20260810 14:43:05.452973 4214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } attempt: 2
I20260810 14:43:05.453928 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.453566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34848)
0810 14:43:05.453749 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.453825 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 14:43:05.454663 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.454431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34850)
0810 14:43:05.454534 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.454589 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 14:43:05.454818 4214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:43783: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
I20260810 14:43:05.473081 4346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } attempt: 2
I20260810 14:43:05.473948 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.473653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34854)
0810 14:43:05.473803 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.473866 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 14:43:05.474665 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.474434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34870)
0810 14:43:05.474545 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.474602 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 14:43:05.474819 4346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:43783: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
W20260810 14:43:05.483754 4355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:05.484061 4355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:05.484122 4355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:05.488349 4355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:43:05.488422 4355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:05.488449 4355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:05.488471 4355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:43:05.488500 4355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:43:05.494140 4355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:44795,127.15.84.61:40803,127.15.84.60:43783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.60.keytab
--principal=kudu/127.15.84.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.60:43783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.84.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:05.495299 4355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:05.496380 4355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:05.505231 4363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:05.505511 4366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:05.505339 4355 server_base.cc:1034] running on GCE node
W20260810 14:43:05.505231 4362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:05.506224 4355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:05.507423 4355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:05.508587 4355 hybrid_clock.cc:648] HybridClock initialized: now 1786372985508564 us; error 39 us; skew 500 ppm
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372985, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:05.511523 4355 init.cc:377] Logged in from keytab as kudu/127.15.84.60@KRBTEST.COM (short username kudu)
I20260810 14:43:05.512512 4355 webserver.cc:460] Webserver started at http://127.15.84.60:42015/ using document root <none> and password file <none>
I20260810 14:43:05.513077 4355 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:05.513180 4355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:05.513420 4355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:43:05.515623 4355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/data/instance:
uuid: "4ff0647bcd7248248ff08d252c49f94b"
format_stamp: "Formatted at 2026-08-10 14:43:05 on dist-test-slave-rmhg"
server_key: "bde07b853e8342accdad2a26ce8f92f8"
server_key_iv: "5eb4c74859e38af1fa3cbbd979f53f57"
server_key_version: "encryptionkey@0"
I20260810 14:43:05.516286 4355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/wal/instance:
uuid: "4ff0647bcd7248248ff08d252c49f94b"
format_stamp: "Formatted at 2026-08-10 14:43:05 on dist-test-slave-rmhg"
server_key: "bde07b853e8342accdad2a26ce8f92f8"
server_key_iv: "5eb4c74859e38af1fa3cbbd979f53f57"
server_key_version: "encryptionkey@0"
I20260810 14:43:05.520231 4355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 14:43:05.522763 4378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:05.524004 4355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:43:05.524123 4355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/wal
uuid: "4ff0647bcd7248248ff08d252c49f94b"
format_stamp: "Formatted at 2026-08-10 14:43:05 on dist-test-slave-rmhg"
server_key: "bde07b853e8342accdad2a26ce8f92f8"
server_key_iv: "5eb4c74859e38af1fa3cbbd979f53f57"
server_key_version: "encryptionkey@0"
I20260810 14:43:05.524250 4355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:05.553988 4214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } attempt: 3
I20260810 14:43:05.554929 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.554576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34878)
0810 14:43:05.554743 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.554826 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 14:43:05.555732 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.555500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34880)
0810 14:43:05.555612 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.555668 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 14:43:05.555866 4214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:43783: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
I20260810 14:43:05.557801 4355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:05.560798 4355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:05.560986 4355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:05.565972 4346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } attempt: 3
I20260810 14:43:05.566761 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.566470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34894)
0810 14:43:05.566604 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.566671 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 14:43:05.567587 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.567326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34900)
0810 14:43:05.567483 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.567530 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260810 14:43:05.567727 4346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:43783: Network error: Client connection negotiation failed: client connection to 127.15.84.60:43783: connect: Connection refused (error 111)
I20260810 14:43:05.568301 4355 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.60:43783
I20260810 14:43:05.568348 4464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.60:43783 every 8 connection(s)
I20260810 14:43:05.569340 4355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/data/info.pb
I20260810 14:43:05.572269 4465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:05.576144 4465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:43:05.578313 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 4355
I20260810 14:43:05.578449 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372819317171-15696-0/minicluster-data/master-2/wal/instance
I20260810 14:43:05.578697 15696 external_mini_cluster.cc:1428] Setting key 97ca51af14a96886e787000ce4a5b8d2
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372985, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.60@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.60@KRBTEST.COM
I20260810 14:43:05.590075 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.578132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.15.84.62:44795)
0810 14:43:05.578291 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.578297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.579236 (+ 939us) server_negotiation.cc:407] Connection header received
0810 14:43:05.580510 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.580513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.580602 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.580676 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.581580 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.582154 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.583153 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.583372 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.583462 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.586119 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.586124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.586127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.586158 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.587651 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.588198 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.588201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.588202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.588280 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.588662 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.588665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.588667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.588925 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.589000 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.589759 (+ 759us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.589900 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:43:05.591144 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.579479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.15.84.60:43783)
0810 14:43:05.580038 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.580051 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.580102 (+ 51us) server_negotiation.cc:407] Connection header received
0810 14:43:05.580280 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.580301 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.580558 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.580751 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.581600 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.582484 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.583324 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.583688 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.583792 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.586715 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.586721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.586733 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.586755 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.588420 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.588889 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.588894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.588896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.588972 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.589273 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.589276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.589278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.589560 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.589755 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.589959 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.590087 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":152,"threads_started":1}
I20260810 14:43:05.591148 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.578542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44795 (local address 127.0.0.1:49534)
0810 14:43:05.579082 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.579129 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:05.580227 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:05.580747 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:05.580752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:05.581102 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:05.581439 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.581454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.582259 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.582262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.583029 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.583039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.583511 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.583515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.583696 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.584274 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:05.584317 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:05.585952 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:05.587766 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.587778 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.587791 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.588105 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.588365 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.588368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.588370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.588557 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.589024 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:05.589032 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:05.589431 (+ 399us) client_negotiation.cc:769] Sending connection context
0810 14:43:05.589653 (+ 222us) client_negotiation.cc:240] Negotiation successful
0810 14:43:05.590012 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":5,"spinlock_wait_cycles":2816,"thread_start_us":139,"threads_started":1}
I20260810 14:43:05.592432 4465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372985, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.60@KRBTEST.COM for kudu/127.15.84.61@KRBTEST.COM
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:05.601101 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.593070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.15.84.61:40803)
0810 14:43:05.593207 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.593212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.593226 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:43:05.593328 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.593331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.593376 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.593439 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.593886 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.594630 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.595653 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.596010 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.596116 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.598290 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.598294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.598295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.598322 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.599585 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.599971 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.599975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.599977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.600056 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.600321 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.600324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.600326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.600509 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.600596 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.600850 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.600962 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:43:05.601101 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.592945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:40803 (local address 127.0.0.1:42064)
0810 14:43:05.593111 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.593124 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:05.593238 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:05.593463 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:05.593465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:05.593637 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:05.593760 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.593769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.594739 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.594742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.595539 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.595548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.596107 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.596110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.596185 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.596653 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:05.596668 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:05.598132 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:05.599693 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.599697 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.599699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.599883 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.600160 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.600162 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.600163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.600238 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.600599 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:05.600602 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:05.600696 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 14:43:05.600783 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 14:43:05.600922 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260810 14:43:05.601951 4465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:43:05.603224 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.593628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.15.84.62:44795)
0810 14:43:05.593752 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.593755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.593768 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:43:05.593827 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.593830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.593873 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.593937 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.594647 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.595384 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.596640 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.596970 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.597081 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.599502 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.599505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.599507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.599531 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.600805 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.601584 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.601586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.601588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.601669 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.601987 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.601990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.601992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.602213 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.602448 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.602996 (+ 548us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.603102 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372985, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.60@KRBTEST.COM for kudu/127.15.84.60@KRBTEST.COM
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.61@KRBTEST.COM
I20260810 14:43:05.609568 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.602488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34912)
0810 14:43:05.602593 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.602606 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:05.602685 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:05.602888 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:05.602891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:05.603040 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:05.603107 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.603113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.603941 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.603944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.604535 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.604556 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.605047 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.605050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.605176 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.605584 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:05.605597 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:05.606690 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:05.608127 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.608130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.608133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.608368 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.608633 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.608636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.608638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.608728 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.609078 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:05.609081 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:05.609255 (+ 174us) client_negotiation.cc:769] Sending connection context
0810 14:43:05.609343 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 14:43:05.609441 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":17}
I20260810 14:43:05.609710 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.602530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.15.84.60:43783)
0810 14:43:05.602623 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.602627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.602639 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:43:05.602771 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.602774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.602812 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.602869 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.603204 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.603859 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.604681 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.604950 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.605038 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.606810 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.606814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.606817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.606840 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.608031 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.608477 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.608480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.608481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.608553 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.608813 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.608816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.608818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.608985 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.609059 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.609422 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.609548 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":37}
I20260810 14:43:05.612221 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.604406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.15.84.61:40803)
0810 14:43:05.604529 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.604533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.604547 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:43:05.604605 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.604608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.604657 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.604720 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.605366 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.605849 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.606570 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.606807 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.606893 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.609228 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.609232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.609234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.609259 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.610789 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.611165 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.611168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.611170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.611255 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.611463 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.611466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.611468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.611675 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.611899 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.611971 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.612079 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 14:43:05.613238 4465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b: Bootstrap starting.
I20260810 14:43:05.615785 4465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:05.616065 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:43:05.616593 4465 log.cc:824] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:05.618728 4465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b: No bootstrap required, opened a new log
I20260810 14:43:05.623082 4465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } }
I20260810 14:43:05.623409 4465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:05.623472 4465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff0647bcd7248248ff08d252c49f94b, State: Initialized, Role: FOLLOWER
I20260810 14:43:05.623968 4465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } }
I20260810 14:43:05.625566 4465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:43:05.627810 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.618084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49556 (local address 127.15.84.62:44795)
0810 14:43:05.618201 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.618205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.618407 (+ 202us) server_negotiation.cc:407] Connection header received
0810 14:43:05.618592 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.618596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.618646 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.618698 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.619664 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.620425 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.621625 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.621957 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.622032 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.623883 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.623887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.623890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.623918 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.625550 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.626246 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.626251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.626252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.626346 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.626665 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.626669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.626670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.626936 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.627035 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.627481 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.627640 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:43:05.628160 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.618468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34914 (local address 127.15.84.60:43783)
0810 14:43:05.618571 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.618574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.619253 (+ 679us) server_negotiation.cc:407] Connection header received
0810 14:43:05.619407 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.619410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.619449 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.619493 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.620350 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.621078 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.622234 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.622540 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.622615 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.624585 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.624589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.624591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.624618 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.626291 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.626770 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.626773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.626775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.626857 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.627156 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.627159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.627161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.627410 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.627498 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.627856 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.627967 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 14:43:05.628444 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.618693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.15.84.61:40803)
0810 14:43:05.618806 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.618810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.618972 (+ 162us) server_negotiation.cc:407] Connection header received
0810 14:43:05.619068 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.619071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.619118 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.619195 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.620002 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.620695 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.621767 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.622093 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.622161 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.624585 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.624589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.624591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.624618 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.626423 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.626953 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.626957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.626959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.627045 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.627373 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.627377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.627378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.627604 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.627731 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.628180 (+ 449us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.628301 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 14:43:05.627997 4485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } } }
I20260810 14:43:05.628584 4485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:43:05.629856 4355 master_runner.cc:186] Error getting master registration for 127.15.84.62:44795: OK, instance_id { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" instance_seqno: 1786372985085730 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:43:05.632318 4355 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 14:43:05.633608 4509 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:43:05.633702 4509 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:43:05.710908 4346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } attempt: 4
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372985, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.61@KRBTEST.COM for kudu/127.15.84.60@KRBTEST.COM
I20260810 14:43:05.720067 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.711553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34928)
0810 14:43:05.711699 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.711716 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:05.711815 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:05.712096 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:05.712099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:05.712331 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:05.712461 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.712468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.713567 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.713571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.714156 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.714167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.714690 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.714692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.714766 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.715245 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:05.715257 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:05.716623 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:05.718589 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.718592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.718594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.718801 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.719103 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.719105 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.719107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.719178 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.719598 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:05.719600 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:05.719691 (+ 91us) client_negotiation.cc:769] Sending connection context
0810 14:43:05.719768 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:43:05.719885 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 14:43:05.720031 4214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } attempt: 4
I20260810 14:43:05.720173 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.711651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.15.84.60:43783)
0810 14:43:05.711800 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.711804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.711823 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:43:05.711881 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.711884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.711935 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.712052 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.712603 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.713443 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.714283 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.714584 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.714694 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.716766 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.716770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.716773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.716803 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.718473 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.718898 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.718902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.718904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.719015 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.719263 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.719266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.719268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.719516 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.719620 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.719844 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.720018 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 14:43:05.724298 4346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c: Bootstrap starting.
Aug 10 14:43:05 dist-test-slave-rmhg krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372985, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for kudu/127.15.84.60@KRBTEST.COM
I20260810 14:43:05.727821 4346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:05.728933 4346 log.cc:824] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:05.729633 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.720594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:43783 (local address 127.0.0.1:34934)
0810 14:43:05.720807 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:05.720824 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:05.720911 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:05.721274 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:05.721278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:05.721505 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:05.721612 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.721618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.722430 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.722434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.723019 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:05.723030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.723469 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.723473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:05.723561 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.724025 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:05.724040 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:05.725787 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:05.727890 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.727894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.727896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.728137 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.728485 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:05.728488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:05.728490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:05.728579 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:05.729031 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:05.729035 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:05.729185 (+ 150us) client_negotiation.cc:769] Sending connection context
0810 14:43:05.729292 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 14:43:05.729455 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260810 14:43:05.729651 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:05.720650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.15.84.60:43783)
0810 14:43:05.721054 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:05.721058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:05.721069 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:43:05.721100 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:05.721103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:05.721139 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:05.721230 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:05.721728 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.722316 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.723139 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:05.723367 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:05.723453 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:05.726242 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:05.726246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:05.726249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:05.726278 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:05.727784 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.728270 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.728273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.728274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.728351 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:05.728667 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:05.728671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:05.728673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:05.728924 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:05.729016 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:05.729368 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 14:43:05.729503 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337}
I20260810 14:43:05.732016 4346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c: No bootstrap required, opened a new log
I20260810 14:43:05.732438 4214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89: Bootstrap starting.
I20260810 14:43:05.734827 4214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:05.735649 4214 log.cc:824] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:05.736636 4346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } }
I20260810 14:43:05.736971 4346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:05.737097 4346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bddb60b720f419bbfb6bd1f23da003c, State: Initialized, Role: FOLLOWER
I20260810 14:43:05.737635 4214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89: No bootstrap required, opened a new log
I20260810 14:43:05.737782 4346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } }
I20260810 14:43:05.738708 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } } }
I20260810 14:43:05.738862 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:43:05.739713 4346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:43:05.741209 4214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } }
I20260810 14:43:05.741446 4214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:05.741508 4214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79192ecdfdf745b6922f4ffd16d6bc89, State: Initialized, Role: FOLLOWER
I20260810 14:43:05.742631 4214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } }
I20260810 14:43:05.744130 4214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 14:43:05.746532 4533 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:43:05.746628 4533 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:43:05.744689 4231 master_runner.cc:186] Error getting master registration for 127.15.84.62:44795: OK, instance_id { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" instance_seqno: 1786372985085730 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:43:05.748872 4231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:43:05.752331 4112 master_runner.cc:263] No leader master found from master 79192ecdfdf745b6922f4ffd16d6bc89
I20260810 14:43:05.752408 4112 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:43:05.753334 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } } }
I20260810 14:43:05.753473 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 14:43:05.756932 4549 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:43:05.757028 4549 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 14:43:05.810340 4264 tablet.cc:2369] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:43:05.812438 4386 tablet.cc:2369] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:43:05.832265 4142 tablet.cc:2369] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:43:06.028017 4485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:43:06.028263 4485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } }
I20260810 14:43:06.029472 4485 leader_election.cc:288] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79192ecdfdf745b6922f4ffd16d6bc89 (127.15.84.62:44795), 6bddb60b720f419bbfb6bd1f23da003c (127.15.84.61:40803)
I20260810 14:43:06.029975 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ff0647bcd7248248ff08d252c49f94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" is_pre_election: true
I20260810 14:43:06.030094 4296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ff0647bcd7248248ff08d252c49f94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bddb60b720f419bbfb6bd1f23da003c" is_pre_election: true
I20260810 14:43:06.030287 4176 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ff0647bcd7248248ff08d252c49f94b in term 0.
I20260810 14:43:06.030397 4296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ff0647bcd7248248ff08d252c49f94b in term 0.
I20260810 14:43:06.030689 4379 leader_election.cc:302] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff0647bcd7248248ff08d252c49f94b, 79192ecdfdf745b6922f4ffd16d6bc89; no voters:
I20260810 14:43:06.030967 4485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:43:06.031059 4485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:43:06.031111 4485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:06.032161 4485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } }
I20260810 14:43:06.032469 4485 leader_election.cc:288] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [CANDIDATE]: Term 1 election: Requested vote from peers 79192ecdfdf745b6922f4ffd16d6bc89 (127.15.84.62:44795), 6bddb60b720f419bbfb6bd1f23da003c (127.15.84.61:40803)
I20260810 14:43:06.032785 4296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ff0647bcd7248248ff08d252c49f94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bddb60b720f419bbfb6bd1f23da003c"
I20260810 14:43:06.032779 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ff0647bcd7248248ff08d252c49f94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79192ecdfdf745b6922f4ffd16d6bc89"
I20260810 14:43:06.032903 4296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:06.032898 4176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:06.033948 4176 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ff0647bcd7248248ff08d252c49f94b in term 1.
I20260810 14:43:06.033985 4296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ff0647bcd7248248ff08d252c49f94b in term 1.
I20260810 14:43:06.034224 4379 leader_election.cc:302] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff0647bcd7248248ff08d252c49f94b, 79192ecdfdf745b6922f4ffd16d6bc89; no voters:
I20260810 14:43:06.034453 4485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:43:06.034646 4485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [term 1 LEADER]: Becoming Leader. State: Replica: 4ff0647bcd7248248ff08d252c49f94b, State: Running, Role: LEADER
I20260810 14:43:06.034967 4485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } }
I20260810 14:43:06.036916 4551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ff0647bcd7248248ff08d252c49f94b. Latest consensus state: current_term: 1 leader_uuid: "4ff0647bcd7248248ff08d252c49f94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } } }
I20260810 14:43:06.037061 4551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:06.037483 4553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:43:06.038332 4553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:43:06.043109 4296 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [term 1 FOLLOWER]: Refusing update from remote peer 4ff0647bcd7248248ff08d252c49f94b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:43:06.043109 4176 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [term 1 FOLLOWER]: Refusing update from remote peer 4ff0647bcd7248248ff08d252c49f94b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:43:06.043702 4551 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:43:06.043879 4485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:43:06.047920 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ff0647bcd7248248ff08d252c49f94b. Latest consensus state: current_term: 1 leader_uuid: "4ff0647bcd7248248ff08d252c49f94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } } }
I20260810 14:43:06.047919 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ff0647bcd7248248ff08d252c49f94b. Latest consensus state: current_term: 1 leader_uuid: "4ff0647bcd7248248ff08d252c49f94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } } }
I20260810 14:43:06.048059 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:43:06.048059 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:43:06.050680 4551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ff0647bcd7248248ff08d252c49f94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } } }
I20260810 14:43:06.050808 4551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:06.051472 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ff0647bcd7248248ff08d252c49f94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } } }
I20260810 14:43:06.051486 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ff0647bcd7248248ff08d252c49f94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } } }
I20260810 14:43:06.051589 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:43:06.051594 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:43:06.051934 4485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ff0647bcd7248248ff08d252c49f94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79192ecdfdf745b6922f4ffd16d6bc89" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44795 } } peers { permanent_uuid: "6bddb60b720f419bbfb6bd1f23da003c" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 40803 } } peers { permanent_uuid: "4ff0647bcd7248248ff08d252c49f94b" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 43783 } } }
I20260810 14:43:06.052047 4485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:06.053630 4553 catalog_manager.cc:1347] Generated new cluster ID: 2e891d3aff554e318c5196f96a5d15e2
I20260810 14:43:06.053691 4553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:43:06.070011 4553 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:43:06.071259 4553 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:43:06.076810 4553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ff0647bcd7248248ff08d252c49f94b: Generated new TSK 0
I20260810 14:43:06.077646 4553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:43:06.747978 4533 catalog_manager.cc:1259] Loaded cluster ID: 2e891d3aff554e318c5196f96a5d15e2
I20260810 14:43:06.748071 4533 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c: loading cluster ID for follower catalog manager: success
I20260810 14:43:06.749878 4533 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c: acquiring CA information for follower catalog manager: success
I20260810 14:43:06.751129 4533 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6bddb60b720f419bbfb6bd1f23da003c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:43:06.758327 4549 catalog_manager.cc:1259] Loaded cluster ID: 2e891d3aff554e318c5196f96a5d15e2
I20260810 14:43:06.758409 4549 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89: loading cluster ID for follower catalog manager: success
I20260810 14:43:06.760094 4549 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89: acquiring CA information for follower catalog manager: success
I20260810 14:43:06.761359 4549 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79192ecdfdf745b6922f4ffd16d6bc89: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:43:07.141872 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 4112
I20260810 14:43:07.157230 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 4231
I20260810 14:43:07.170755 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 4355
2026-08-10T14:43:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3579 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:43:07 dist-test-slave-rmhg krb5kdc[4579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:43:07 dist-test-slave-rmhg krb5kdc[4579](info): set up 2 sockets
Aug 10 14:43:07 dist-test-slave-rmhg krb5kdc[4579](info): commencing operation
krb5kdc: starting...
W20260810 14:43:08.324779 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:43:08 dist-test-slave-rmhg krb5kdc[4579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:43:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:43:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:43:08.458312 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40099
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:34053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:40099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:08.576359 4595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:08.576660 4595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:08.576715 4595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:08.580843 4595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:43:08.580926 4595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:08.580946 4595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:08.580962 4595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:43:08.580976 4595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:43:08.586553 4595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:40099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:08.587687 4595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:08.588722 4595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:08.597489 4608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:08.597489 4602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:08.597512 4605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:08.597870 4595 server_base.cc:1034] running on GCE node
I20260810 14:43:08.598464 4595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:08.599567 4595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:08.600740 4595 hybrid_clock.cc:648] HybridClock initialized: now 1786372988600713 us; error 45 us; skew 500 ppm
Aug 10 14:43:08 dist-test-slave-rmhg krb5kdc[4579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372988, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:08.606590 4595 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:43:08.607653 4595 webserver.cc:460] Webserver started at http://127.15.84.62:41411/ using document root <none> and password file <none>
I20260810 14:43:08.608283 4595 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:08.608359 4595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:08.608572 4595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:43:08.610805 4595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "530c008be2ae4012be64856ae744e92e"
format_stamp: "Formatted at 2026-08-10 14:43:08 on dist-test-slave-rmhg"
server_key: "c2c072b601dce90b875e93cdb320fa51"
server_key_iv: "18ec8551dbc2e19755ef879876700941"
server_key_version: "encryptionkey@0"
I20260810 14:43:08.611425 4595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "530c008be2ae4012be64856ae744e92e"
format_stamp: "Formatted at 2026-08-10 14:43:08 on dist-test-slave-rmhg"
server_key: "c2c072b601dce90b875e93cdb320fa51"
server_key_iv: "18ec8551dbc2e19755ef879876700941"
server_key_version: "encryptionkey@0"
I20260810 14:43:08.615481 4595 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:43:08.618075 4618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:08.619320 4595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:43:08.619439 4595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "530c008be2ae4012be64856ae744e92e"
format_stamp: "Formatted at 2026-08-10 14:43:08 on dist-test-slave-rmhg"
server_key: "c2c072b601dce90b875e93cdb320fa51"
server_key_iv: "18ec8551dbc2e19755ef879876700941"
server_key_version: "encryptionkey@0"
I20260810 14:43:08.619539 4595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:08.641184 4595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:08.644451 4595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:08.644701 4595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:08.652092 4595 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:40099
I20260810 14:43:08.652110 4698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:40099 every 8 connection(s)
I20260810 14:43:08.653102 4595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:43:08.656144 4699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:08.656563 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 4595
I20260810 14:43:08.656694 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:43:08.656970 15696 external_mini_cluster.cc:1428] Setting key e8ea589c2bf6c321ad74b9e7990ad07b
I20260810 14:43:08.663815 4699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e: Bootstrap starting.
I20260810 14:43:08.666718 4699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e: Neither blocks nor log segments found. Creating new log.
Aug 10 14:43:08 dist-test-slave-rmhg krb5kdc[4579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:08.667605 4699 log.cc:824] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:08.670074 4699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e: No bootstrap required, opened a new log
I20260810 14:43:08.672659 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:08.658421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.15.84.62:40099)
0810 14:43:08.658934 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:08.658959 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:08.659006 (+ 47us) server_negotiation.cc:407] Connection header received
0810 14:43:08.659739 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:08.659765 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:08.660220 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:08.660615 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:08.661339 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:08.662900 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:08.663838 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:08.664549 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:08.664720 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:08.667393 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:08.667399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:08.667418 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:08.667453 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:08.669463 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:08.669893 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:08.669899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:08.669905 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:08.670048 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:08.670327 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:08.670331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:08.670332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:08.670892 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:08.671087 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:08.671302 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 14:43:08.671611 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":120,"threads_started":1}
I20260810 14:43:08.674580 4699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530c008be2ae4012be64856ae744e92e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40099 } }
I20260810 14:43:08.674886 4699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:08.675009 4699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530c008be2ae4012be64856ae744e92e, State: Initialized, Role: FOLLOWER
I20260810 14:43:08.675546 4699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530c008be2ae4012be64856ae744e92e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40099 } }
I20260810 14:43:08.675725 4699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:43:08.675799 4699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:43:08.675908 4699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:08.676034 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:43:08.677145 4699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530c008be2ae4012be64856ae744e92e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40099 } }
I20260810 14:43:08.677584 4699 leader_election.cc:302] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 530c008be2ae4012be64856ae744e92e; no voters:
I20260810 14:43:08.677948 4699 leader_election.cc:288] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:43:08.678049 4709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:43:08.678278 4709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [term 1 LEADER]: Becoming Leader. State: Replica: 530c008be2ae4012be64856ae744e92e, State: Running, Role: LEADER
I20260810 14:43:08.678645 4709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530c008be2ae4012be64856ae744e92e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40099 } }
I20260810 14:43:08.679251 4699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:43:08.680444 4714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 530c008be2ae4012be64856ae744e92e. Latest consensus state: current_term: 1 leader_uuid: "530c008be2ae4012be64856ae744e92e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530c008be2ae4012be64856ae744e92e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40099 } } }
I20260810 14:43:08.680451 4711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "530c008be2ae4012be64856ae744e92e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "530c008be2ae4012be64856ae744e92e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40099 } } }
I20260810 14:43:08.680611 4714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:08.680611 4711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:08.681012 4726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:43:08.684600 4726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:43:08.692991 4726 catalog_manager.cc:1347] Generated new cluster ID: 21a820cf20eb44c3a6a0818b37059c35
I20260810 14:43:08.693085 4726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:43:08.726985 4726 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:43:08.728037 4726 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:43:08.744944 4726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 530c008be2ae4012be64856ae744e92e: Generated new TSK 0
I20260810 14:43:08.745894 4726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:43:08.783366 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 4595
2026-08-10T14:43:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1611 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:43:08 dist-test-slave-rmhg krb5kdc[4748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:43:08 dist-test-slave-rmhg krb5kdc[4748](info): set up 2 sockets
Aug 10 14:43:08 dist-test-slave-rmhg krb5kdc[4748](info): commencing operation
krb5kdc: starting...
W20260810 14:43:09.943113 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:43:10 dist-test-slave-rmhg krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:43:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:43:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:43:10.076223 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:44379
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:44379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:10.192857 4769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:10.193195 4769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:10.193272 4769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:10.197440 4769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:43:10.197522 4769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:10.197548 4769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:10.197571 4769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:43:10.197590 4769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:43:10.203192 4769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:44379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:44379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:10.204408 4769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:10.205677 4769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:10.214758 4776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:10.214809 4769 server_base.cc:1034] running on GCE node
W20260810 14:43:10.214813 4781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:10.215014 4778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:10.215637 4769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:10.216845 4769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:10.218022 4769 hybrid_clock.cc:648] HybridClock initialized: now 1786372990217977 us; error 60 us; skew 500 ppm
Aug 10 14:43:10 dist-test-slave-rmhg krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372990, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:10.221052 4769 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:43:10.222074 4769 webserver.cc:460] Webserver started at http://127.15.84.62:43541/ using document root <none> and password file <none>
I20260810 14:43:10.222692 4769 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:10.222767 4769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:10.222983 4769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:43:10.225201 4769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "ec3fabf5eb5640618d60ab5a672276c7"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "f9ef31186f69f891c8377339742cce63"
server_key_iv: "301eaef70eb47d512e47aad183ad37df"
server_key_version: "encryptionkey@0"
I20260810 14:43:10.225798 4769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "ec3fabf5eb5640618d60ab5a672276c7"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "f9ef31186f69f891c8377339742cce63"
server_key_iv: "301eaef70eb47d512e47aad183ad37df"
server_key_version: "encryptionkey@0"
I20260810 14:43:10.229663 4769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:43:10.232195 4794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:10.233551 4769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:43:10.233704 4769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "ec3fabf5eb5640618d60ab5a672276c7"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "f9ef31186f69f891c8377339742cce63"
server_key_iv: "301eaef70eb47d512e47aad183ad37df"
server_key_version: "encryptionkey@0"
I20260810 14:43:10.233811 4769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:10.251282 4769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:10.254802 4769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:10.255007 4769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:10.262399 4769 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:44379
I20260810 14:43:10.262671 4884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:44379 every 8 connection(s)
I20260810 14:43:10.263528 4769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:43:10.264081 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 4769
I20260810 14:43:10.264226 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:43:10.264490 15696 external_mini_cluster.cc:1428] Setting key d3c51b324543d2bbe21d59135e06e449
I20260810 14:43:10.266872 4885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:10.273654 4885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Bootstrap starting.
Aug 10 14:43:10 dist-test-slave-rmhg krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:10.276299 4885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:10.277235 4885 log.cc:824] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:10.279047 4890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:10.266004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.15.84.62:44379)
0810 14:43:10.266426 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:10.266444 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:10.266482 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:43:10.267178 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:10.267197 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:10.267560 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:10.267851 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:10.268740 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:10.269842 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:10.270998 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:10.271488 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:10.271603 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:10.274414 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:10.274420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:10.274433 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:10.274465 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:10.276018 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:10.276512 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:10.276517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:10.276522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:10.276641 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:10.276912 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:10.276914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:10.276915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:10.277409 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:10.277564 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:10.277934 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 14:43:10.278268 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260810 14:43:10.279476 4885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: No bootstrap required, opened a new log
I20260810 14:43:10.283252 4885 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } }
I20260810 14:43:10.283504 4885 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:10.283556 4885 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3fabf5eb5640618d60ab5a672276c7, State: Initialized, Role: FOLLOWER
I20260810 14:43:10.284029 4885 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } }
I20260810 14:43:10.284170 4885 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:43:10.284237 4885 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:43:10.284340 4885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:10.285420 4885 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } }
I20260810 14:43:10.285791 4885 leader_election.cc:302] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec3fabf5eb5640618d60ab5a672276c7; no voters:
I20260810 14:43:10.286164 4885 leader_election.cc:288] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:43:10.286268 4893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:43:10.286502 4893 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 1 LEADER]: Becoming Leader. State: Replica: ec3fabf5eb5640618d60ab5a672276c7, State: Running, Role: LEADER
I20260810 14:43:10.286890 4893 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } }
I20260810 14:43:10.287456 4885 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:43:10.288525 4895 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec3fabf5eb5640618d60ab5a672276c7. Latest consensus state: current_term: 1 leader_uuid: "ec3fabf5eb5640618d60ab5a672276c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } } }
I20260810 14:43:10.288661 4895 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:10.288954 4894 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec3fabf5eb5640618d60ab5a672276c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } } }
I20260810 14:43:10.289038 4894 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:10.289062 4911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:43:10.291904 4911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:43:10.298045 4911 catalog_manager.cc:1347] Generated new cluster ID: 2f5ecd296892408ebe15e06b3d075f71
I20260810 14:43:10.298137 4911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:43:10.310256 4911 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:43:10.311702 4911 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:43:10.324445 4911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Generated new TSK 0
I20260810 14:43:10.325489 4911 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:43:10.359776 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:44379
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:10.484617 4934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:10.484918 4934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:10.484963 4934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:10.489253 4934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:10.489331 4934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:10.489395 4934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:43:10.494943 4934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:10.496037 4934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:10.497035 4934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:10.507259 4941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:10.507261 4946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:10.507261 4942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:10.507498 4934 server_base.cc:1034] running on GCE node
I20260810 14:43:10.508008 4934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:10.508705 4934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:10.509869 4934 hybrid_clock.cc:648] HybridClock initialized: now 1786372990509846 us; error 37 us; skew 500 ppm
Aug 10 14:43:10 dist-test-slave-rmhg krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372990, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:10.512714 4934 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:43:10.513686 4934 webserver.cc:460] Webserver started at http://127.15.84.1:35157/ using document root <none> and password file <none>
I20260810 14:43:10.514248 4934 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:10.514317 4934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:10.514528 4934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:43:10.516598 4934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "c3275a4aca1f46e78bce20d3dee61493"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "5a278a84b8b87c72ae96cda45c3937c9"
server_key_iv: "b3d71b8783c8e1b6db064f8836864b65"
server_key_version: "encryptionkey@0"
I20260810 14:43:10.517210 4934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "c3275a4aca1f46e78bce20d3dee61493"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "5a278a84b8b87c72ae96cda45c3937c9"
server_key_iv: "b3d71b8783c8e1b6db064f8836864b65"
server_key_version: "encryptionkey@0"
I20260810 14:43:10.520992 4934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 14:43:10.523506 4963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:10.524770 4934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:43:10.524895 4934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "c3275a4aca1f46e78bce20d3dee61493"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "5a278a84b8b87c72ae96cda45c3937c9"
server_key_iv: "b3d71b8783c8e1b6db064f8836864b65"
server_key_version: "encryptionkey@0"
I20260810 14:43:10.524994 4934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:10.542938 4934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:10.546325 4934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:10.546522 4934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:10.547160 4934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:10.548064 4934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:10.548116 4934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:10.548173 4934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:10.548197 4934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:10.556951 4934 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:43507
I20260810 14:43:10.557014 5139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:43507 every 8 connection(s)
I20260810 14:43:10.557929 4934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:43:10.559208 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 4934
I20260810 14:43:10.559329 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:43:10.559612 15696 external_mini_cluster.cc:1428] Setting key 700da0ae9292565884bce78e76131de3
Aug 10 14:43:10 dist-test-slave-rmhg krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372990, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:43:10.571975 4890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:10.559796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:51979 (local address 127.15.84.62:44379)
0810 14:43:10.559960 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:10.559965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:10.560851 (+ 886us) server_negotiation.cc:407] Connection header received
0810 14:43:10.562422 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:10.562426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:10.562484 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:10.562576 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:10.563832 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:10.564384 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:10.565659 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:10.565898 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:10.565990 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:10.568664 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:10.568668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:10.568670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:10.568698 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:10.570094 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:10.570626 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:10.570629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:10.570631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:10.570711 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:10.570988 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:10.570990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:10.570991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:10.571165 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:10.571238 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:10.571725 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 14:43:10.571846 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
I20260810 14:43:10.572875 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:10.560217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:51979)
0810 14:43:10.560716 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:10.560762 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:10.562194 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:10.562735 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:10.562743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:10.563157 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:10.563677 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:10.563692 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:10.564534 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:10.564537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:10.565537 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:10.565548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:10.566062 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:10.566067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:10.566230 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:10.566790 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:10.566808 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:10.568437 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:10.570212 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:10.570219 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:10.570234 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:10.570527 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:10.570784 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:10.570786 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:10.570787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:10.570923 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:10.571237 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:10.571242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:10.571503 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 14:43:10.571665 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 14:43:10.571967 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":138,"threads_started":1}
I20260810 14:43:10.574184 5141 heartbeater.cc:347] Connected to a master server at 127.15.84.62:44379
I20260810 14:43:10.574453 5141 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:10.574968 5141 heartbeater.cc:511] Master 127.15.84.62:44379 requested a full tablet report, sending...
I20260810 14:43:10.576687 4821 ts_manager.cc:194] Registered new tserver with Master: c3275a4aca1f46e78bce20d3dee61493 (127.15.84.1:43507)
I20260810 14:43:10.578446 4821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:51979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:43:10.590293 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:44379
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:10.707463 5152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:10.707748 5152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:10.707794 5152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:10.711830 5152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:10.711901 5152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:10.711974 5152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:43:10.717490 5152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:10.718645 5152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:10.719733 5152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:10.729532 5165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:10.729466 5163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:10.729542 5162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:10.729918 5152 server_base.cc:1034] running on GCE node
I20260810 14:43:10.730376 5152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:10.731061 5152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:10.732234 5152 hybrid_clock.cc:648] HybridClock initialized: now 1786372990732211 us; error 36 us; skew 500 ppm
Aug 10 14:43:10 dist-test-slave-rmhg krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372990, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:10.735052 5152 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:43:10.736001 5152 webserver.cc:460] Webserver started at http://127.15.84.2:37423/ using document root <none> and password file <none>
I20260810 14:43:10.736564 5152 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:10.736634 5152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:10.736850 5152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:43:10.738999 5152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "c97ef34a1e344744aff36f73c4c2f5cc"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "8c3b755b796082447971ceab470f93c2"
server_key_iv: "7ca93f023af9189b54a837b22281db2e"
server_key_version: "encryptionkey@0"
I20260810 14:43:10.739552 5152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "c97ef34a1e344744aff36f73c4c2f5cc"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "8c3b755b796082447971ceab470f93c2"
server_key_iv: "7ca93f023af9189b54a837b22281db2e"
server_key_version: "encryptionkey@0"
I20260810 14:43:10.743583 5152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:43:10.746217 5176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:10.747484 5152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:43:10.747603 5152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "c97ef34a1e344744aff36f73c4c2f5cc"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "8c3b755b796082447971ceab470f93c2"
server_key_iv: "7ca93f023af9189b54a837b22281db2e"
server_key_version: "encryptionkey@0"
I20260810 14:43:10.747713 5152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:10.763479 5152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:10.766563 5152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:10.766764 5152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:10.767373 5152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:10.768231 5152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:10.768282 5152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:10.768337 5152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:10.768361 5152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:10.777374 5152 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:33407
I20260810 14:43:10.777448 5340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:33407 every 8 connection(s)
I20260810 14:43:10.778380 5152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:43:10.788017 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 5152
Aug 10 14:43:10 dist-test-slave-rmhg krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372990, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:10.788139 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:43:10.788383 15696 external_mini_cluster.cc:1428] Setting key a6115f71534aa86e535be4816d25b9e8
I20260810 14:43:10.792781 4890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:10.780296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:48965 (local address 127.15.84.62:44379)
0810 14:43:10.780441 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:10.780445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:10.781333 (+ 888us) server_negotiation.cc:407] Connection header received
0810 14:43:10.782757 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:10.782761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:10.782822 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:10.782897 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:10.784164 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:10.784712 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:10.785916 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:10.786139 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:10.786243 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:10.788883 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:10.788888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:10.788891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:10.788942 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:10.790346 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:10.790940 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:10.790943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:10.790945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:10.791029 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:10.791431 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:10.791434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:10.791435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:10.791675 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:10.791765 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:10.792425 (+ 660us) server_negotiation.cc:299] Negotiation successful
0810 14:43:10.792607 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 14:43:10.793776 5344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:10.780657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:48965)
0810 14:43:10.781141 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:10.781235 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:10.782538 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:10.783100 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:10.783108 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:10.783490 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:10.784011 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:10.784026 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:10.784834 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:10.784837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:10.785796 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:10.785806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:10.786286 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:10.786290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:10.786459 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:10.787028 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:10.787045 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:10.788703 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:10.790473 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:10.790479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:10.790494 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:10.790827 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:10.791112 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:10.791115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:10.791116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:10.791309 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:10.791783 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:10.791790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:10.792139 (+ 349us) client_negotiation.cc:769] Sending connection context
0810 14:43:10.792335 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 14:43:10.792663 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":116,"threads_started":1}
I20260810 14:43:10.794919 5342 heartbeater.cc:347] Connected to a master server at 127.15.84.62:44379
I20260810 14:43:10.795159 5342 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:10.795666 5342 heartbeater.cc:511] Master 127.15.84.62:44379 requested a full tablet report, sending...
I20260810 14:43:10.796738 4821 ts_manager.cc:194] Registered new tserver with Master: c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407)
I20260810 14:43:10.797353 4821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:48965
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:43:10.818396 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:44379
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:10.936902 5349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:10.937266 5349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:10.937321 5349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:10.941470 5349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:10.941541 5349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:10.941615 5349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:43:10.947211 5349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:10.948366 5349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:10.949503 5349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:10.959460 5358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:10.959460 5357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:10.959563 5349 server_base.cc:1034] running on GCE node
W20260810 14:43:10.959465 5361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:10.960242 5349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:10.960960 5349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:10.962128 5349 hybrid_clock.cc:648] HybridClock initialized: now 1786372990962093 us; error 47 us; skew 500 ppm
Aug 10 14:43:10 dist-test-slave-rmhg krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372990, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:10.964833 5349 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:43:10.965843 5349 webserver.cc:460] Webserver started at http://127.15.84.3:37283/ using document root <none> and password file <none>
I20260810 14:43:10.966411 5349 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:10.966481 5349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:10.966679 5349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:43:10.968801 5349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "0c32ebc1e12641008326a3653ffee651"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "f80da2e6a5afb10db301ae9615f2f81d"
server_key_iv: "bb5ecffe6505035f7c518e5fcb6fded4"
server_key_version: "encryptionkey@0"
I20260810 14:43:10.969416 5349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "0c32ebc1e12641008326a3653ffee651"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "f80da2e6a5afb10db301ae9615f2f81d"
server_key_iv: "bb5ecffe6505035f7c518e5fcb6fded4"
server_key_version: "encryptionkey@0"
I20260810 14:43:10.973305 5349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:43:10.975728 5371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:10.976982 5349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:43:10.977093 5349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "0c32ebc1e12641008326a3653ffee651"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "f80da2e6a5afb10db301ae9615f2f81d"
server_key_iv: "bb5ecffe6505035f7c518e5fcb6fded4"
server_key_version: "encryptionkey@0"
I20260810 14:43:10.977252 5349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:10.995771 5349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:10.998653 5349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:10.998839 5349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:10.999461 5349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:11.000377 5349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:11.000427 5349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:11.000470 5349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:11.000499 5349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:11.009473 5349 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:39833
I20260810 14:43:11.009496 5548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:39833 every 8 connection(s)
I20260810 14:43:11.010445 5349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:43:11.016539 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 5349
I20260810 14:43:11.016675 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:43:11.016919 15696 external_mini_cluster.cc:1428] Setting key d22788cc8f859b27992b84bc3fd8d237
Aug 10 14:43:11 dist-test-slave-rmhg krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372990, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:11.024190 4890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.012275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:37771 (local address 127.15.84.62:44379)
0810 14:43:11.012409 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:11.012413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:11.013365 (+ 952us) server_negotiation.cc:407] Connection header received
0810 14:43:11.014814 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:11.014817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:11.014874 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:11.014954 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:11.016184 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.016746 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.018064 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.018284 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.018374 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.020898 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:11.020902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:11.020904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:11.020928 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:11.022192 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:11.022692 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:11.022695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:11.022696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:11.022773 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:11.023177 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:11.023181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:11.023182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:11.023378 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:11.023462 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:11.023964 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 14:43:11.024065 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:43:11.025031 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.012672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:37771)
0810 14:43:11.013210 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:11.013258 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:11.014601 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:11.015148 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:11.015156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:11.015534 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:11.016037 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.016053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.016866 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.016869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.017943 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.017954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.018429 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.018433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.018617 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.019195 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:11.019212 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:11.020726 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:11.022303 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:11.022309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:11.022325 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:11.022592 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:11.022858 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:11.022861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:11.022863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:11.023070 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:11.023511 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:11.023517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:11.023770 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 14:43:11.023932 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 14:43:11.024204 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":129,"threads_started":1}
I20260810 14:43:11.026116 5550 heartbeater.cc:347] Connected to a master server at 127.15.84.62:44379
I20260810 14:43:11.026340 5550 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:11.026844 5550 heartbeater.cc:511] Master 127.15.84.62:44379 requested a full tablet report, sending...
I20260810 14:43:11.027825 4821 ts_manager.cc:194] Registered new tserver with Master: 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833)
I20260810 14:43:11.028404 4821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:37771
I20260810 14:43:11.030604 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:43:11.039728 4890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.032584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.15.84.62:44379)
0810 14:43:11.032748 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:11.032753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:11.032882 (+ 129us) server_negotiation.cc:407] Connection header received
0810 14:43:11.032997 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:11.033000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:11.033041 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:11.033111 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:11.033897 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.034428 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.035282 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.035489 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.035555 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.036814 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:11.036817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:11.036820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:11.036844 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:11.038186 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:11.038589 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:11.038592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:11.038594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:11.038702 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:11.038969 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:11.038970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:11.038971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:11.039182 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:11.039280 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:11.039462 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 14:43:11.039556 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 14:43:11.042697 4821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:43:11.045897 4821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:43:11.061369 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.054977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43507 (local address 127.0.0.1:46246)
0810 14:43:11.055621 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:11.055636 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:11.055722 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:11.056887 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:11.056894 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:11.056920 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:11.057073 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.057079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.058654 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.058657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.059833 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.059844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.060931 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.060934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.061011 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.061029 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 14:43:11.061072 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 14:43:11.061141 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"spinlock_wait_cycles":7040,"thread_start_us":139,"threads_started":1}
I20260810 14:43:11.061390 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.055856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.15.84.2:33407)
0810 14:43:11.056220 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:11.056233 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:11.056259 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:43:11.056346 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:11.056352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:11.056582 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:11.056744 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:11.057236 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.058439 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.060123 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.060760 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.060862 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.061009 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:11.061113 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 14:43:11.061244 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":120,"threads_started":1}
I20260810 14:43:11.061609 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.055510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.15.84.3:39833)
0810 14:43:11.056451 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:11.056462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:11.056482 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:43:11.056550 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:11.056554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:11.056732 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:11.056865 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:11.057262 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.058031 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.059634 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.060513 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.060632 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.060801 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:11.061345 (+ 544us) server_negotiation.cc:299] Negotiation successful
0810 14:43:11.061451 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":115,"threads_started":1}
I20260810 14:43:11.061668 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.055959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46246 (local address 127.15.84.1:43507)
0810 14:43:11.056368 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:11.056386 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:11.056409 (+ 23us) server_negotiation.cc:407] Connection header received
0810 14:43:11.056489 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:11.056495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:11.056703 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:11.057261 (+ 558us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:11.057430 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.058490 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.059983 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.060807 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.061251 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.061402 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:11.061446 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 14:43:11.061528 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":90,"threads_started":1}
I20260810 14:43:11.061252 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.054847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:39833 (local address 127.0.0.1:59522)
0810 14:43:11.055235 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:11.055293 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:11.055427 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:11.057025 (+ 1598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:11.057028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:11.057042 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:11.057116 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.057121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.058171 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.058175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.059495 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.059506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.060658 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.060662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.060770 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.060819 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 14:43:11.060929 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 14:43:11.061065 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260810 14:43:11.062094 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.055641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:33407 (local address 127.0.0.1:39694)
0810 14:43:11.055937 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:11.055949 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:11.056028 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:11.056944 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:11.056947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:11.056966 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:11.057077 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.057083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.058590 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.058594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.060002 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.060011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.060891 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.060895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.060984 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.061010 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:43:11.061085 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 14:43:11.061141 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":292,"thread_start_us":94,"threads_started":1}
I20260810 14:43:11.065363 5029 tablet_service.cc:1467] Processing CreateTablet for tablet 2dda00949631441ba3fedbf1909def46 (DEFAULT_TABLE table=test-table [id=0f2bba00b7344f8cbd5c7b256004d04f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:11.065361 5443 tablet_service.cc:1467] Processing CreateTablet for tablet 2dda00949631441ba3fedbf1909def46 (DEFAULT_TABLE table=test-table [id=0f2bba00b7344f8cbd5c7b256004d04f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:11.066556 5029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dda00949631441ba3fedbf1909def46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:11.066560 5443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dda00949631441ba3fedbf1909def46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:11.067631 5245 tablet_service.cc:1467] Processing CreateTablet for tablet 2dda00949631441ba3fedbf1909def46 (DEFAULT_TABLE table=test-table [id=0f2bba00b7344f8cbd5c7b256004d04f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:11.068495 5245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dda00949631441ba3fedbf1909def46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:11.074910 5577 tablet_bootstrap.cc:492] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc: Bootstrap starting.
I20260810 14:43:11.075587 5574 tablet_bootstrap.cc:492] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Bootstrap starting.
I20260810 14:43:11.077368 5574 tablet_bootstrap.cc:654] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:11.077387 5577 tablet_bootstrap.cc:654] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:11.077544 5576 tablet_bootstrap.cc:492] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651: Bootstrap starting.
I20260810 14:43:11.078169 5574 log.cc:824] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:11.078217 5577 log.cc:824] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:11.079627 5576 tablet_bootstrap.cc:654] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:11.080431 5574 tablet_bootstrap.cc:492] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: No bootstrap required, opened a new log
I20260810 14:43:11.080650 5574 ts_tablet_manager.cc:1397] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 14:43:11.080899 5576 log.cc:824] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:11.081306 5577 tablet_bootstrap.cc:492] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc: No bootstrap required, opened a new log
I20260810 14:43:11.081563 5577 ts_tablet_manager.cc:1397] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 14:43:11.083066 5576 tablet_bootstrap.cc:492] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651: No bootstrap required, opened a new log
I20260810 14:43:11.083279 5576 ts_tablet_manager.cc:1397] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:43:11.084568 5577 raft_consensus.cc:348] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } }
I20260810 14:43:11.084807 5577 raft_consensus.cc:374] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:11.084861 5577 raft_consensus.cc:729] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97ef34a1e344744aff36f73c4c2f5cc, State: Initialized, Role: FOLLOWER
I20260810 14:43:11.084957 5574 raft_consensus.cc:348] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } }
I20260810 14:43:11.085248 5574 raft_consensus.cc:374] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:11.085316 5574 raft_consensus.cc:729] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3275a4aca1f46e78bce20d3dee61493, State: Initialized, Role: FOLLOWER
I20260810 14:43:11.085367 5577 consensus_queue.cc:260] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } }
I20260810 14:43:11.085796 5574 consensus_queue.cc:260] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } }
I20260810 14:43:11.086127 5342 heartbeater.cc:503] Master 127.15.84.62:44379 was elected leader, sending a full tablet report...
I20260810 14:43:11.086421 5577 ts_tablet_manager.cc:1428] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 14:43:11.086771 5574 ts_tablet_manager.cc:1428] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 14:43:11.087185 5141 heartbeater.cc:503] Master 127.15.84.62:44379 was elected leader, sending a full tablet report...
I20260810 14:43:11.087036 5576 raft_consensus.cc:348] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } }
I20260810 14:43:11.087885 5576 raft_consensus.cc:374] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:11.087945 5576 raft_consensus.cc:729] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c32ebc1e12641008326a3653ffee651, State: Initialized, Role: FOLLOWER
I20260810 14:43:11.088500 5576 consensus_queue.cc:260] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } }
I20260810 14:43:11.089212 5550 heartbeater.cc:503] Master 127.15.84.62:44379 was elected leader, sending a full tablet report...
I20260810 14:43:11.089622 5576 ts_tablet_manager.cc:1428] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 14:43:11.255172 5587 raft_consensus.cc:482] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:43:11.255401 5587 raft_consensus.cc:504] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } }
I20260810 14:43:11.256596 5587 leader_election.cc:288] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833), c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407)
I20260810 14:43:11.260886 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.257321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:33407 (local address 127.15.84.1:41525)
0810 14:43:11.257705 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:11.257720 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:11.257810 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:11.258025 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:11.258028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:11.258045 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:11.258165 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.258172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.258878 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.258881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.259792 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.259802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.260444 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.260447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.260542 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.260584 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 14:43:11.260646 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 14:43:11.260712 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260810 14:43:11.260901 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.257442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:41525 (local address 127.15.84.2:33407)
0810 14:43:11.257598 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:11.257602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:11.257797 (+ 195us) server_negotiation.cc:407] Connection header received
0810 14:43:11.257889 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:11.257893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:11.257938 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:11.257994 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:11.258276 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.258764 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.259918 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.260310 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.260391 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.260460 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:11.260690 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 14:43:11.260773 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:43:11.260924 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.256970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:58505 (local address 127.15.84.3:39833)
0810 14:43:11.257109 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:11.257113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:11.257288 (+ 175us) server_negotiation.cc:407] Connection header received
0810 14:43:11.257441 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:11.257445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:11.257504 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:11.257595 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:11.258193 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.258753 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.259937 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.260341 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.260445 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.260509 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:11.260744 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 14:43:11.260820 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:43:11.260919 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.256866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:39833 (local address 127.15.84.1:58505)
0810 14:43:11.257195 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:11.257210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:11.257329 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:11.257634 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:11.257638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:11.257662 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:11.257803 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.257810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.258870 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.258874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.259800 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.259810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.260459 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.260463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.260607 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.260626 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 14:43:11.260703 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:43:11.260780 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":92,"thread_start_us":112,"threads_started":1}
W20260810 14:43:11.261390 5551 tablet.cc:2369] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:43:11.261726 5475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dda00949631441ba3fedbf1909def46" candidate_uuid: "c3275a4aca1f46e78bce20d3dee61493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c32ebc1e12641008326a3653ffee651" is_pre_election: true
I20260810 14:43:11.261731 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dda00949631441ba3fedbf1909def46" candidate_uuid: "c3275a4aca1f46e78bce20d3dee61493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" is_pre_election: true
I20260810 14:43:11.262039 5271 raft_consensus.cc:2454] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3275a4aca1f46e78bce20d3dee61493 in term 0.
I20260810 14:43:11.262038 5475 raft_consensus.cc:2454] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3275a4aca1f46e78bce20d3dee61493 in term 0.
I20260810 14:43:11.262533 4964 leader_election.cc:302] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3275a4aca1f46e78bce20d3dee61493, c97ef34a1e344744aff36f73c4c2f5cc; no voters:
I20260810 14:43:11.262823 5587 raft_consensus.cc:2781] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:43:11.262921 5587 raft_consensus.cc:482] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:43:11.262970 5587 raft_consensus.cc:3037] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:11.264297 5587 raft_consensus.cc:504] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } }
I20260810 14:43:11.264703 5587 leader_election.cc:288] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 1 election: Requested vote from peers 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833), c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407)
I20260810 14:43:11.265094 5475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dda00949631441ba3fedbf1909def46" candidate_uuid: "c3275a4aca1f46e78bce20d3dee61493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c32ebc1e12641008326a3653ffee651"
I20260810 14:43:11.265214 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dda00949631441ba3fedbf1909def46" candidate_uuid: "c3275a4aca1f46e78bce20d3dee61493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97ef34a1e344744aff36f73c4c2f5cc"
I20260810 14:43:11.265266 5475 raft_consensus.cc:3037] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:11.265342 5271 raft_consensus.cc:3037] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:11.266580 5475 raft_consensus.cc:2454] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3275a4aca1f46e78bce20d3dee61493 in term 1.
I20260810 14:43:11.266580 5271 raft_consensus.cc:2454] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3275a4aca1f46e78bce20d3dee61493 in term 1.
I20260810 14:43:11.266887 4964 leader_election.cc:302] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3275a4aca1f46e78bce20d3dee61493, c97ef34a1e344744aff36f73c4c2f5cc; no voters:
I20260810 14:43:11.267068 5587 raft_consensus.cc:2781] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:43:11.267331 5587 raft_consensus.cc:686] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 1 LEADER]: Becoming Leader. State: Replica: c3275a4aca1f46e78bce20d3dee61493, State: Running, Role: LEADER
I20260810 14:43:11.267704 5587 consensus_queue.cc:237] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } }
I20260810 14:43:11.271242 4821 catalog_manager.cc:5495] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3275a4aca1f46e78bce20d3dee61493 (127.15.84.1). New cstate: current_term: 1 leader_uuid: "c3275a4aca1f46e78bce20d3dee61493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } health_report { overall_health: UNKNOWN } } }
W20260810 14:43:11.279543 5343 tablet.cc:2369] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:43:11.299528 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.296322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46256 (local address 127.15.84.1:43507)
0810 14:43:11.296435 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:11.296439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:11.296452 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:43:11.296568 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:11.296570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:11.296625 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:11.296732 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:43:11.296925 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.297475 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.298285 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.298495 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.298569 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.298815 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:43:11.299245 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:43:11.299334 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:11.299365 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 14:43:11.299417 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 14:43:11.305709 5475 raft_consensus.cc:1261] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651 [term 1 FOLLOWER]: Refusing update from remote peer c3275a4aca1f46e78bce20d3dee61493: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:43:11.305727 5271 raft_consensus.cc:1261] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc [term 1 FOLLOWER]: Refusing update from remote peer c3275a4aca1f46e78bce20d3dee61493: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:43:11.306464 5587 consensus_queue.cc:1035] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:43:11.306689 5593 consensus_queue.cc:1035] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:43:11.309300 5143 tablet.cc:2369] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:43:11.316845 5601 mvcc.cc:205] Tried to move back new op lower bound from 7316983772380401664 to 7316983772235468800. Current Snapshot: MvccSnapshot[applied={T|T < 7316983772380401664}]
I20260810 14:43:11.323746 4819 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:43:11.323930 4819 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:43:11.326529 4819 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Sending DeleteTablet for 3 replicas of tablet 2dda00949631441ba3fedbf1909def46
I20260810 14:43:11.327425 5029 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dda00949631441ba3fedbf1909def46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:11 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:46246
I20260810 14:43:11.327507 5245 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dda00949631441ba3fedbf1909def46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:11 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:39694
I20260810 14:43:11.327636 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dda00949631441ba3fedbf1909def46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:11 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:59522
I20260810 14:43:11.327970 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 4934
I20260810 14:43:11.328014 5610 tablet_replica.cc:330] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc: stopping tablet replica
I20260810 14:43:11.328814 5610 raft_consensus.cc:2229] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:43:11.329058 5612 tablet_replica.cc:330] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651: stopping tablet replica
I20260810 14:43:11.329427 5612 raft_consensus.cc:2229] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:43:11.329617 5610 raft_consensus.cc:2258] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:43:11.329749 5612 raft_consensus.cc:2258] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:43:11.335172 5610 ts_tablet_manager.cc:1905] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:43:11.337941 5610 ts_tablet_manager.cc:1918] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:43:11.338050 5610 log.cc:1196] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/2dda00949631441ba3fedbf1909def46
I20260810 14:43:11.338426 5610 ts_tablet_manager.cc:1939] T 2dda00949631441ba3fedbf1909def46 P c97ef34a1e344744aff36f73c4c2f5cc: Deleting consensus metadata
I20260810 14:43:11.339818 4795 catalog_manager.cc:4841] TS c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407): tablet 2dda00949631441ba3fedbf1909def46 (table test-table [id=0f2bba00b7344f8cbd5c7b256004d04f]) successfully deleted
I20260810 14:43:11.340483 5612 ts_tablet_manager.cc:1905] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:43:11.343444 5612 ts_tablet_manager.cc:1918] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:43:11.343588 5612 log.cc:1196] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/2dda00949631441ba3fedbf1909def46
I20260810 14:43:11.343982 5612 ts_tablet_manager.cc:1939] T 2dda00949631441ba3fedbf1909def46 P 0c32ebc1e12641008326a3653ffee651: Deleting consensus metadata
I20260810 14:43:11.345515 4795 catalog_manager.cc:4841] TS 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833): tablet 2dda00949631441ba3fedbf1909def46 (table test-table [id=0f2bba00b7344f8cbd5c7b256004d04f]) successfully deleted
W20260810 14:43:11.347070 4797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:43507 (error 108)
I20260810 14:43:11.347134 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 5152
I20260810 14:43:11.360430 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.357912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43507 (local address 127.0.0.1:46264)
0810 14:43:11.360197 (+ 2285us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:11.360312 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2150}
W20260810 14:43:11.360746 4797 catalog_manager.cc:4584] TS c3275a4aca1f46e78bce20d3dee61493 (127.15.84.1:43507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2dda00949631441ba3fedbf1909def46: Network error: Client connection negotiation failed: client connection to 127.15.84.1:43507: connect: Connection refused (error 111)
I20260810 14:43:11.366385 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 5349
I20260810 14:43:11.379585 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 4769
I20260810 14:43:11.392840 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:44379
--webserver_interface=127.15.84.62
--webserver_port=43541
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:44379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:11.510594 5615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:11.510941 5615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:11.510990 5615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:11.515240 5615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:43:11.515323 5615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:11.515343 5615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:11.515358 5615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:43:11.515374 5615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:43:11.521057 5615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:44379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:44379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=43541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:11.522267 5615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:11.523348 5615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:11.537873 5630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:11.537873 5624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:11.537988 5615 server_base.cc:1034] running on GCE node
W20260810 14:43:11.537873 5627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:11.538791 5615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:11.540050 5615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:11.541229 5615 hybrid_clock.cc:648] HybridClock initialized: now 1786372991541201 us; error 42 us; skew 500 ppm
Aug 10 14:43:11 dist-test-slave-rmhg krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372991, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:11.544085 5615 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:43:11.545120 5615 webserver.cc:460] Webserver started at http://127.15.84.62:43541/ using document root <none> and password file <none>
I20260810 14:43:11.545770 5615 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:11.545861 5615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:11.550112 5615 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:43:11.552421 5642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:11.553784 5615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:43:11.553915 5615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "ec3fabf5eb5640618d60ab5a672276c7"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "f9ef31186f69f891c8377339742cce63"
server_key_iv: "301eaef70eb47d512e47aad183ad37df"
server_key_version: "encryptionkey@0"
I20260810 14:43:11.554541 5615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:11.576861 5615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:11.580439 5615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:11.580657 5615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:11.588425 5615 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:44379
I20260810 14:43:11.588721 5727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:44379 every 8 connection(s)
I20260810 14:43:11.589740 5615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:43:11.590425 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 5615
I20260810 14:43:11.590901 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:43507
--local_ip_for_outbound_sockets=127.15.84.1
--tserver_master_addrs=127.15.84.62:44379
--webserver_port=35157
--webserver_interface=127.15.84.1
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:43:11.595273 5728 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:43:11.597829 5728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Bootstrap starting.
I20260810 14:43:11.622270 5728 log.cc:824] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:11.633495 5728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:43:11.633908 5728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Bootstrap complete.
I20260810 14:43:11.638240 5728 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } }
I20260810 14:43:11.638664 5728 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3fabf5eb5640618d60ab5a672276c7, State: Initialized, Role: FOLLOWER
I20260810 14:43:11.639190 5728 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } }
I20260810 14:43:11.639328 5728 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:43:11.639400 5728 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:43:11.639500 5728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:43:11.641592 5728 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } }
I20260810 14:43:11.641950 5728 leader_election.cc:302] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec3fabf5eb5640618d60ab5a672276c7; no voters:
I20260810 14:43:11.642287 5728 leader_election.cc:288] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 14:43:11.642395 5736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:43:11.642581 5736 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 2 LEADER]: Becoming Leader. State: Replica: ec3fabf5eb5640618d60ab5a672276c7, State: Running, Role: LEADER
I20260810 14:43:11.642916 5736 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } }
I20260810 14:43:11.643659 5728 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:43:11.644115 5736 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec3fabf5eb5640618d60ab5a672276c7. Latest consensus state: current_term: 2 leader_uuid: "ec3fabf5eb5640618d60ab5a672276c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } } }
I20260810 14:43:11.644238 5736 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:11.644570 5737 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ec3fabf5eb5640618d60ab5a672276c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } } }
I20260810 14:43:11.644651 5741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:43:11.644783 5737 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:11.651240 5741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:43:11.652027 5741 catalog_manager.cc:1259] Loaded cluster ID: 2f5ecd296892408ebe15e06b3d075f71
I20260810 14:43:11.652105 5741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:43:11.653687 5741 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 14:43:11.658624 5741 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 14:43:11.665045 5741 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:43:11.666067 5741 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Loaded TSK: 0
I20260810 14:43:11.667431 5741 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:43:11.728971 5733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:11.729337 5733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:11.729411 5733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:11.733763 5733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:11.733846 5733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:11.733935 5733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:43:11.739622 5733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:43507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=35157
--tserver_master_addrs=127.15.84.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:11.740778 5733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:11.741900 5733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:11.752589 5780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:11.752575 5786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:11.752573 5781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:11.753422 5733 server_base.cc:1034] running on GCE node
I20260810 14:43:11.753904 5733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:11.754652 5733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:11.755827 5733 hybrid_clock.cc:648] HybridClock initialized: now 1786372991755793 us; error 54 us; skew 500 ppm
Aug 10 14:43:11 dist-test-slave-rmhg krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372991, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:11.758977 5733 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:43:11.760016 5733 webserver.cc:460] Webserver started at http://127.15.84.1:35157/ using document root <none> and password file <none>
I20260810 14:43:11.760608 5733 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:11.760699 5733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:11.765136 5733 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:43:11.767663 5796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:11.769270 5733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 14:43:11.769418 5733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "c3275a4aca1f46e78bce20d3dee61493"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "5a278a84b8b87c72ae96cda45c3937c9"
server_key_iv: "b3d71b8783c8e1b6db064f8836864b65"
server_key_version: "encryptionkey@0"
I20260810 14:43:11.770051 5733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:11.784969 5733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:11.788293 5733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:11.788496 5733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:11.789188 5733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:11.790500 5808 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 14:43:11.795006 5733 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 14:43:11.795087 5733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260810 14:43:11.795145 5733 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 14:43:11.797009 5733 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 14:43:11.797080 5733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 14:43:11.797425 5808 tablet_bootstrap.cc:492] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Bootstrap starting.
I20260810 14:43:11.808413 5733 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:43507
I20260810 14:43:11.808475 5976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:43507 every 8 connection(s)
I20260810 14:43:11.809897 5733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:43:11.810066 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 5733
I20260810 14:43:11.810546 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:33407
--local_ip_for_outbound_sockets=127.15.84.2
--tserver_master_addrs=127.15.84.62:44379
--webserver_port=37423
--webserver_interface=127.15.84.2
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:43:11.821229 5808 log.cc:824] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Log is configured to *not* fsync() on all Append() calls
Aug 10 14:43:11 dist-test-slave-rmhg krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372991, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:11.829946 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.812946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:47837 (local address 127.15.84.62:44379)
0810 14:43:11.813557 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:11.813580 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:11.814190 (+ 610us) server_negotiation.cc:407] Connection header received
0810 14:43:11.815699 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:11.815725 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:11.816066 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:11.816400 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:11.817719 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.818796 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.820026 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.820524 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.820685 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.823733 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:11.823740 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:11.823759 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:11.823798 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:11.826054 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:11.826742 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:11.826749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:11.826755 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:11.826926 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:11.827335 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:11.827338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:11.827340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:11.827868 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:11.828011 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:11.828718 (+ 707us) server_negotiation.cc:299] Negotiation successful
0810 14:43:11.829048 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":138,"threads_started":1}
I20260810 14:43:11.829967 5982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.812828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:47837)
0810 14:43:11.813962 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:11.814026 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:11.815190 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:11.816494 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:11.816503 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:11.816945 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:11.817529 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.817545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.818966 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.818969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.819894 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.819905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.820671 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.820675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.820874 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.821510 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:11.821529 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:11.823462 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:11.826202 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:11.826208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:11.826225 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:11.826589 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:11.827025 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:11.827028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:11.827031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:11.827227 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:11.827987 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:11.827994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:11.828341 (+ 347us) client_negotiation.cc:769] Sending connection context
0810 14:43:11.828535 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 14:43:11.828905 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":145,"threads_started":1}
I20260810 14:43:11.830547 5808 tablet_bootstrap.cc:492] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:43:11.830992 5808 tablet_bootstrap.cc:492] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Bootstrap complete.
I20260810 14:43:11.832427 5808 ts_tablet_manager.cc:1397] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.019s
I20260810 14:43:11.833294 5977 heartbeater.cc:347] Connected to a master server at 127.15.84.62:44379
I20260810 14:43:11.833549 5977 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:11.834142 5977 heartbeater.cc:511] Master 127.15.84.62:44379 requested a full tablet report, sending...
I20260810 14:43:11.835041 5808 raft_consensus.cc:348] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } }
I20260810 14:43:11.835492 5808 raft_consensus.cc:729] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3275a4aca1f46e78bce20d3dee61493, State: Initialized, Role: FOLLOWER
I20260810 14:43:11.835978 5808 consensus_queue.cc:260] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } }
I20260810 14:43:11.837087 5808 ts_tablet_manager.cc:1428] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260810 14:43:11.837072 5670 ts_manager.cc:194] Registered new tserver with Master: c3275a4aca1f46e78bce20d3dee61493 (127.15.84.1:43507)
I20260810 14:43:11.840953 5670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:47837
I20260810 14:43:11.843849 5977 heartbeater.cc:503] Master 127.15.84.62:44379 was elected leader, sending a full tablet report...
Aug 10 14:43:11 dist-test-slave-rmhg krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372991, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for kudu/127.15.84.1@KRBTEST.COM
I20260810 14:43:11.851626 5987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.840084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43507 (local address 127.0.0.1:46280)
0810 14:43:11.840555 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:11.840605 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:11.840743 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:11.841472 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:11.841478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:11.841943 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:11.842090 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.842101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.843101 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.843106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.843776 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:11.843788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.845297 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.845301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:11.845423 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.846061 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:11.846077 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:11.847751 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:11.849925 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:11.849936 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:11.849941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:11.850185 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:11.850496 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:11.850498 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:11.850499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:11.850576 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:11.851068 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:11.851073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:11.851188 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 14:43:11.851313 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 14:43:11.851484 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":131,"threads_started":1}
I20260810 14:43:11.852008 5988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:11.840178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.15.84.1:43507)
0810 14:43:11.840924 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:11.840937 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:11.840964 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:43:11.841040 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:11.841045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:11.841301 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:11.841441 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:11.842220 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.842977 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.843902 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:11.844428 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:11.845302 (+ 874us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:11.847908 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:11.847919 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:11.847933 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:11.847963 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:11.849806 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:11.850307 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:11.850314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:11.850315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:11.850414 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:11.850646 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:11.850650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:11.850651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:11.850970 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:11.851092 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:11.851409 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 14:43:11.851591 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":640,"thread_start_us":102,"threads_started":1}
I20260810 14:43:11.853027 5874 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dda00949631441ba3fedbf1909def46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:11 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:46280
I20260810 14:43:11.853960 5989 tablet_replica.cc:330] stopping tablet replica
I20260810 14:43:11.854277 5989 raft_consensus.cc:2229] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:43:11.854533 5989 raft_consensus.cc:2258] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:43:11.855543 5989 ts_tablet_manager.cc:1905] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:43:11.859434 5989 ts_tablet_manager.cc:1918] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:43:11.859534 5989 log.cc:1196] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal/wals/2dda00949631441ba3fedbf1909def46
I20260810 14:43:11.859776 5989 ts_tablet_manager.cc:1939] T 2dda00949631441ba3fedbf1909def46 P c3275a4aca1f46e78bce20d3dee61493: Deleting consensus metadata
I20260810 14:43:11.861547 5647 catalog_manager.cc:4841] TS c3275a4aca1f46e78bce20d3dee61493 (127.15.84.1:43507): tablet 2dda00949631441ba3fedbf1909def46 (table test-table [id=0f2bba00b7344f8cbd5c7b256004d04f]) successfully deleted
W20260810 14:43:11.938198 5981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:11.938493 5981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:11.938556 5981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:11.942934 5981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:11.943004 5981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:11.943089 5981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:43:11.948849 5981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:33407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=37423
--tserver_master_addrs=127.15.84.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:11.949950 5981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:11.950970 5981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:11.960619 6000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:11.960659 5998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:11.960644 5997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:11.961014 5981 server_base.cc:1034] running on GCE node
I20260810 14:43:11.961469 5981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:11.962093 5981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:11.963267 5981 hybrid_clock.cc:648] HybridClock initialized: now 1786372991963247 us; error 37 us; skew 500 ppm
Aug 10 14:43:11 dist-test-slave-rmhg krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372991, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:11.965886 5981 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:43:11.966768 5981 webserver.cc:460] Webserver started at http://127.15.84.2:37423/ using document root <none> and password file <none>
I20260810 14:43:11.967319 5981 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:11.967402 5981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:11.971215 5981 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:43:11.973296 6008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:11.974524 5981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:43:11.974627 5981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "c97ef34a1e344744aff36f73c4c2f5cc"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "8c3b755b796082447971ceab470f93c2"
server_key_iv: "7ca93f023af9189b54a837b22281db2e"
server_key_version: "encryptionkey@0"
I20260810 14:43:11.975157 5981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:11.992385 5981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:11.995456 5981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:11.995666 5981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:11.996302 5981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:11.997221 5981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:11.997272 5981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:11.997327 5981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:11.997354 5981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:12.006656 5981 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:33407
I20260810 14:43:12.006686 6179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:33407 every 8 connection(s)
I20260810 14:43:12.007782 5981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:43:12.011162 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 5981
I20260810 14:43:12.011708 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:39833
--local_ip_for_outbound_sockets=127.15.84.3
--tserver_master_addrs=127.15.84.62:44379
--webserver_port=37283
--webserver_interface=127.15.84.3
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:43:12 dist-test-slave-rmhg krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372991, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:12.023363 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.009729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:52319 (local address 127.15.84.62:44379)
0810 14:43:12.009872 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:12.009876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:12.010779 (+ 903us) server_negotiation.cc:407] Connection header received
0810 14:43:12.012553 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:12.012556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:12.012613 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:12.012693 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:12.014244 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.015031 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.016206 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.016588 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.016703 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.019520 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:12.019525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:12.019528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:12.019564 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:12.021173 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:12.021700 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:12.021704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:12.021706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:12.021807 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:12.022141 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:12.022144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:12.022146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:12.022399 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:12.022510 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:12.023010 (+ 500us) server_negotiation.cc:299] Negotiation successful
0810 14:43:12.023180 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:43:12.024190 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.010115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:52319)
0810 14:43:12.010610 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:12.010657 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:12.012300 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:12.012918 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:12.012928 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:12.013460 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:12.014086 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:12.014102 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.015145 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.015148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:12.016067 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:12.016077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.016765 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.016769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:12.016928 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.017553 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:12.017572 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:12.019312 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:12.021293 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:12.021299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:12.021314 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:12.021597 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:12.021892 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:12.021895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:12.021896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:12.022044 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:12.022493 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:12.022498 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:12.022767 (+ 269us) client_negotiation.cc:769] Sending connection context
0810 14:43:12.022931 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 14:43:12.023228 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
I20260810 14:43:12.025671 6180 heartbeater.cc:347] Connected to a master server at 127.15.84.62:44379
I20260810 14:43:12.025938 6180 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:12.026467 6180 heartbeater.cc:511] Master 127.15.84.62:44379 requested a full tablet report, sending...
I20260810 14:43:12.027655 5670 ts_manager.cc:194] Registered new tserver with Master: c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407)
I20260810 14:43:12.028455 5670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:52319
W20260810 14:43:12.129244 6184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:12.129506 6184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:12.129549 6184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:12.133509 6184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:12.133569 6184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:12.133625 6184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:43:12.138815 6184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:39833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=37283
--tserver_master_addrs=127.15.84.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:12.139813 6184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:12.140805 6184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:12.150779 6196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:12.150801 6194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:12.150841 6192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:12.150969 6184 server_base.cc:1034] running on GCE node
I20260810 14:43:12.151432 6184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:12.152097 6184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:12.153221 6184 hybrid_clock.cc:648] HybridClock initialized: now 1786372992153200 us; error 33 us; skew 500 ppm
Aug 10 14:43:12 dist-test-slave-rmhg krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372992, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:12.155932 6184 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:43:12.156901 6184 webserver.cc:460] Webserver started at http://127.15.84.3:37283/ using document root <none> and password file <none>
I20260810 14:43:12.157579 6184 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:12.157666 6184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:12.161697 6184 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:43:12.163872 6207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:12.165189 6184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:43:12.165303 6184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "0c32ebc1e12641008326a3653ffee651"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "f80da2e6a5afb10db301ae9615f2f81d"
server_key_iv: "bb5ecffe6505035f7c518e5fcb6fded4"
server_key_version: "encryptionkey@0"
I20260810 14:43:12.165856 6184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:12.188400 6184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:12.191565 6184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:12.191742 6184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:12.192315 6184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:12.193141 6184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:12.193234 6184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:12.193285 6184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:12.193305 6184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:12.202234 6184 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:39833
I20260810 14:43:12.202261 6385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:39833 every 8 connection(s)
I20260810 14:43:12.203351 6184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:43:12.209838 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 6184
Aug 10 14:43:12 dist-test-slave-rmhg krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372992, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:12.217969 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.205327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:35531 (local address 127.15.84.62:44379)
0810 14:43:12.205477 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:12.205481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:12.206409 (+ 928us) server_negotiation.cc:407] Connection header received
0810 14:43:12.207763 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:12.207766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:12.207816 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:12.207893 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:12.209122 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.209753 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.211206 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.211488 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.211567 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.214399 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:12.214402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:12.214404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:12.214420 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:12.215708 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:12.216218 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:12.216221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:12.216223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:12.216318 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:12.216719 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:12.216723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:12.216725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:12.216953 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:12.217085 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:12.217705 (+ 620us) server_negotiation.cc:299] Negotiation successful
0810 14:43:12.217830 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56,"spinlock_wait_cycles":1920}
I20260810 14:43:12.218851 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.210561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47110 (local address 127.15.84.62:44379)
0810 14:43:12.210828 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:12.210832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:12.210981 (+ 149us) server_negotiation.cc:407] Connection header received
0810 14:43:12.211121 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:12.211123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:12.211166 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:12.211255 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:12.212131 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.212644 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.213719 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.214012 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.214098 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.215551 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:12.215555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:12.215558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:12.215605 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:12.217145 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:12.217585 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:12.217587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:12.217588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:12.217682 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:12.217961 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:12.217964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:12.217966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:12.218212 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:12.218417 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:12.218592 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 14:43:12.218702 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"spinlock_wait_cycles":2944,"thread_start_us":83,"threads_started":1}
I20260810 14:43:12.218979 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.205746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:35531)
0810 14:43:12.206235 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:12.206281 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:12.207552 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:12.208058 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:12.208065 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:12.208460 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:12.208977 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:12.208992 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.209871 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.209874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:12.211097 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:12.211109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.211665 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.211671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:12.211929 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.212536 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:12.212553 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:12.214243 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:12.215834 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:12.215840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:12.215855 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:12.216119 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:12.216420 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:12.216424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:12.216426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:12.216627 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:12.217049 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:12.217054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:12.217428 (+ 374us) client_negotiation.cc:769] Sending connection context
0810 14:43:12.217646 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 14:43:12.218026 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":119,"threads_started":1}
I20260810 14:43:12.220158 6386 heartbeater.cc:347] Connected to a master server at 127.15.84.62:44379
I20260810 14:43:12.220398 6386 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:12.220963 6386 heartbeater.cc:511] Master 127.15.84.62:44379 requested a full tablet report, sending...
I20260810 14:43:12.222025 5670 ts_manager.cc:194] Registered new tserver with Master: 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833)
I20260810 14:43:12.222571 5670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:35531
I20260810 14:43:12.232950 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:43:12.237406 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.233710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.15.84.62:44379)
0810 14:43:12.233840 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:12.233844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:12.234122 (+ 278us) server_negotiation.cc:407] Connection header received
0810 14:43:12.234272 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:12.234275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:12.234393 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:12.234486 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:43:12.234703 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.235236 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.236043 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.236273 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.236343 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.236617 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:43:12.237058 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:43:12.237188 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:12.237221 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 14:43:12.237288 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:43:12.238137 5670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:43:12.241578 5670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:43:12.250036 5874 tablet_service.cc:1467] Processing CreateTablet for tablet a546d54506a4494d91ff67c7edcd11cd (DEFAULT_TABLE table=test-table [id=a331111502b9495ab66dc1f105951131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:12.250803 5874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a546d54506a4494d91ff67c7edcd11cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:12.253660 5808 tablet_bootstrap.cc:492] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493: Bootstrap starting.
I20260810 14:43:12.254441 5987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.248899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:39833 (local address 127.0.0.1:59530)
0810 14:43:12.249014 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:12.249030 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:12.249114 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:12.250213 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:12.250216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:12.250233 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:12.250346 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:12.250353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.251570 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.251573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:12.252742 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:12.252754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.253981 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.253985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:12.254087 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.254158 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 14:43:12.254215 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 14:43:12.254304 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":4}
I20260810 14:43:12.254585 5808 tablet_bootstrap.cc:654] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:12.254725 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.249132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:33407 (local address 127.0.0.1:39702)
0810 14:43:12.249434 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:12.249447 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:12.249532 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:12.250601 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:12.250604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:12.250623 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:12.250713 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:12.250719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.252065 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.252069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:12.253060 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:12.253068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.254166 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.254170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:12.254280 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.254307 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 14:43:12.254357 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 14:43:12.254428 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":125,"threads_started":1}
I20260810 14:43:12.255038 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.249111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.15.84.3:39833)
0810 14:43:12.249664 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:12.249678 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:12.249704 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:43:12.249795 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:12.249802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:12.250051 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:12.250213 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:12.250491 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.251435 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.252888 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.253837 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.254523 (+ 686us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.254699 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:12.254769 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 14:43:12.254872 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":151,"threads_started":1}
I20260810 14:43:12.255406 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.249660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39702 (local address 127.15.84.2:33407)
0810 14:43:12.250052 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:12.250065 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:12.250090 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:43:12.250184 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:12.250190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:12.250454 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:12.250611 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:12.250811 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.251917 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.253203 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.254024 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.254892 (+ 868us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.255011 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:12.255057 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 14:43:12.255290 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":163,"threads_started":1}
I20260810 14:43:12.256225 5808 tablet_bootstrap.cc:492] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493: No bootstrap required, opened a new log
I20260810 14:43:12.256318 5808 ts_tablet_manager.cc:1397] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:43:12.256809 5808 raft_consensus.cc:348] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:12.256942 5808 raft_consensus.cc:374] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:12.256989 5808 raft_consensus.cc:729] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3275a4aca1f46e78bce20d3dee61493, State: Initialized, Role: FOLLOWER
I20260810 14:43:12.258415 6083 tablet_service.cc:1467] Processing CreateTablet for tablet a546d54506a4494d91ff67c7edcd11cd (DEFAULT_TABLE table=test-table [id=a331111502b9495ab66dc1f105951131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:12.258864 6285 tablet_service.cc:1467] Processing CreateTablet for tablet a546d54506a4494d91ff67c7edcd11cd (DEFAULT_TABLE table=test-table [id=a331111502b9495ab66dc1f105951131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:12.259442 6083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a546d54506a4494d91ff67c7edcd11cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:12.259817 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a546d54506a4494d91ff67c7edcd11cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:12.261291 5808 consensus_queue.cc:260] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:12.261633 5808 ts_tablet_manager.cc:1428] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260810 14:43:12.267001 6410 tablet_bootstrap.cc:492] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc: Bootstrap starting.
I20260810 14:43:12.267227 6411 tablet_bootstrap.cc:492] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651: Bootstrap starting.
I20260810 14:43:12.269431 6411 tablet_bootstrap.cc:654] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:12.269644 6410 tablet_bootstrap.cc:654] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:12.270506 6410 log.cc:824] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:12.270800 6411 log.cc:824] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:12.272452 6410 tablet_bootstrap.cc:492] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc: No bootstrap required, opened a new log
I20260810 14:43:12.272468 6411 tablet_bootstrap.cc:492] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651: No bootstrap required, opened a new log
I20260810 14:43:12.272646 6410 ts_tablet_manager.cc:1397] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:43:12.272646 6411 ts_tablet_manager.cc:1397] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 14:43:12.275818 6411 raft_consensus.cc:348] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:12.275816 6410 raft_consensus.cc:348] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:12.276095 6410 raft_consensus.cc:374] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:12.276094 6411 raft_consensus.cc:374] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:12.276157 6410 raft_consensus.cc:729] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97ef34a1e344744aff36f73c4c2f5cc, State: Initialized, Role: FOLLOWER
I20260810 14:43:12.276162 6411 raft_consensus.cc:729] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c32ebc1e12641008326a3653ffee651, State: Initialized, Role: FOLLOWER
I20260810 14:43:12.276568 6411 consensus_queue.cc:260] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:12.276582 6410 consensus_queue.cc:260] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:12.277259 6386 heartbeater.cc:503] Master 127.15.84.62:44379 was elected leader, sending a full tablet report...
I20260810 14:43:12.277271 6180 heartbeater.cc:503] Master 127.15.84.62:44379 was elected leader, sending a full tablet report...
I20260810 14:43:12.277637 6411 ts_tablet_manager.cc:1428] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260810 14:43:12.278663 6410 ts_tablet_manager.cc:1428] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260810 14:43:12.311493 5980 tablet.cc:2369] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:43:12.441824 6417 raft_consensus.cc:482] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:43:12.442373 6417 raft_consensus.cc:504] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:12.443730 6417 leader_election.cc:288] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833), c3275a4aca1f46e78bce20d3dee61493 (127.15.84.1:43507)
I20260810 14:43:12.448134 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.444194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:53677 (local address 127.15.84.3:39833)
0810 14:43:12.444323 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:12.444327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:12.444342 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:43:12.444402 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:12.444405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:12.444456 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:12.444546 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:12.444900 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.445773 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.447114 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.447525 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.447621 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.447700 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:12.447879 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 14:43:12.447958 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:43:12.448217 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.443956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:39833 (local address 127.15.84.2:53677)
0810 14:43:12.444181 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:12.444196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:12.444287 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:12.444594 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:12.444598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:12.444620 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:12.444754 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:12.444761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.445907 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.445910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:12.446981 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:12.446992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.447637 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.447641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:12.447753 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.447776 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:43:12.447842 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 14:43:12.447920 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":22}
I20260810 14:43:12.448997 6317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a546d54506a4494d91ff67c7edcd11cd" candidate_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c32ebc1e12641008326a3653ffee651" is_pre_election: true
I20260810 14:43:12.449200 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.443968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43507 (local address 127.15.84.2:33877)
0810 14:43:12.444865 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:12.444891 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:12.444981 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:12.445281 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:12.445284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:12.445301 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:12.445418 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:12.445423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.446316 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.446320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:12.447608 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:12.447622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.448761 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.448764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:12.448862 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.448888 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:43:12.448949 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 14:43:12.449012 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":416,"thread_start_us":174,"threads_started":1}
I20260810 14:43:12.449412 6317 raft_consensus.cc:2454] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c97ef34a1e344744aff36f73c4c2f5cc in term 0.
I20260810 14:43:12.449460 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.444106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:33877 (local address 127.15.84.1:43507)
0810 14:43:12.444378 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:12.444383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:12.444975 (+ 592us) server_negotiation.cc:407] Connection header received
0810 14:43:12.445068 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:12.445071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:12.445134 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:12.445263 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:12.445525 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.446213 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.447775 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.448629 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.448755 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.448956 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:12.449009 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 14:43:12.449100 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":98,"threads_started":1}
I20260810 14:43:12.449994 6009 leader_election.cc:302] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c32ebc1e12641008326a3653ffee651, c97ef34a1e344744aff36f73c4c2f5cc; no voters:
I20260810 14:43:12.450047 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a546d54506a4494d91ff67c7edcd11cd" candidate_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3275a4aca1f46e78bce20d3dee61493" is_pre_election: true
I20260810 14:43:12.450325 6417 raft_consensus.cc:2781] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:43:12.450361 5907 raft_consensus.cc:2454] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c97ef34a1e344744aff36f73c4c2f5cc in term 0.
I20260810 14:43:12.450416 6417 raft_consensus.cc:482] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:43:12.450462 6417 raft_consensus.cc:3037] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:12.451566 6417 raft_consensus.cc:504] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:12.451926 6417 leader_election.cc:288] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [CANDIDATE]: Term 1 election: Requested vote from peers 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833), c3275a4aca1f46e78bce20d3dee61493 (127.15.84.1:43507)
I20260810 14:43:12.452337 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a546d54506a4494d91ff67c7edcd11cd" candidate_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3275a4aca1f46e78bce20d3dee61493"
I20260810 14:43:12.452492 5907 raft_consensus.cc:3037] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:12.453299 6317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a546d54506a4494d91ff67c7edcd11cd" candidate_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c32ebc1e12641008326a3653ffee651"
I20260810 14:43:12.453425 6317 raft_consensus.cc:3037] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:12.453958 5907 raft_consensus.cc:2454] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c97ef34a1e344744aff36f73c4c2f5cc in term 1.
I20260810 14:43:12.454294 6011 leader_election.cc:302] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3275a4aca1f46e78bce20d3dee61493, c97ef34a1e344744aff36f73c4c2f5cc; no voters:
W20260810 14:43:12.454391 6387 tablet.cc:2369] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:43:12.454648 6317 raft_consensus.cc:2454] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c97ef34a1e344744aff36f73c4c2f5cc in term 1.
I20260810 14:43:12.454862 6417 raft_consensus.cc:2781] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:43:12.455101 6417 raft_consensus.cc:686] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [term 1 LEADER]: Becoming Leader. State: Replica: c97ef34a1e344744aff36f73c4c2f5cc, State: Running, Role: LEADER
I20260810 14:43:12.455436 6417 consensus_queue.cc:237] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:12.458778 5670 catalog_manager.cc:5495] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2). New cstate: current_term: 1 leader_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } health_report { overall_health: UNKNOWN } } }
I20260810 14:43:12.496833 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.493652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39708 (local address 127.15.84.2:33407)
0810 14:43:12.493789 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:12.493793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:12.493806 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:43:12.493903 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:12.493906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:12.493962 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:12.494048 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:43:12.494234 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.494746 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.495504 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:12.495761 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:12.495857 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:12.496080 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:43:12.496524 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:43:12.496613 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:12.496643 (+ 30us) server_negotiation.cc:299] Negotiation successful
0810 14:43:12.496694 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:43:12.502942 5907 raft_consensus.cc:1261] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Refusing update from remote peer c97ef34a1e344744aff36f73c4c2f5cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:43:12.502948 6317 raft_consensus.cc:1261] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651 [term 1 FOLLOWER]: Refusing update from remote peer c97ef34a1e344744aff36f73c4c2f5cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:43:12.503697 6417 consensus_queue.cc:1035] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:43:12.503903 6422 consensus_queue.cc:1035] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:43:12.508905 6182 tablet.cc:2369] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:43:12.510568 6426 mvcc.cc:205] Tried to move back new op lower bound from 7316983777284116480 to 7316983777100361728. Current Snapshot: MvccSnapshot[applied={T|T < 7316983777284116480}]
I20260810 14:43:12.512776 6434 mvcc.cc:205] Tried to move back new op lower bound from 7316983777284116480 to 7316983777100361728. Current Snapshot: MvccSnapshot[applied={T|T < 7316983777284116480}]
I20260810 14:43:12.520939 5670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:43:12.521144 5670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:43:12.523866 5670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Sending DeleteTablet for 3 replicas of tablet a546d54506a4494d91ff67c7edcd11cd
I20260810 14:43:12.524629 5874 tablet_service.cc:1514] Processing DeleteTablet for tablet a546d54506a4494d91ff67c7edcd11cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:12 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:46280
I20260810 14:43:12.524730 6083 tablet_service.cc:1514] Processing DeleteTablet for tablet a546d54506a4494d91ff67c7edcd11cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:12 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:39702
I20260810 14:43:12.524726 6285 tablet_service.cc:1514] Processing DeleteTablet for tablet a546d54506a4494d91ff67c7edcd11cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:12 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:59530
I20260810 14:43:12.525302 6448 tablet_replica.cc:330] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651: stopping tablet replica
I20260810 14:43:12.525347 6444 tablet_replica.cc:330] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493: stopping tablet replica
I20260810 14:43:12.525451 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 5733
I20260810 14:43:12.525661 6448 raft_consensus.cc:2229] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:43:12.526021 6448 raft_consensus.cc:2258] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:43:12.534853 6445 tablet_replica.cc:330] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc: stopping tablet replica
I20260810 14:43:12.535328 6445 raft_consensus.cc:2229] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:43:12.535892 6445 raft_consensus.cc:2258] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:43:12.538997 6448 ts_tablet_manager.cc:1905] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:43:12.540308 6445 ts_tablet_manager.cc:1905] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:43:12.542389 6448 ts_tablet_manager.cc:1918] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:43:12.542527 6448 log.cc:1196] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/a546d54506a4494d91ff67c7edcd11cd
I20260810 14:43:12.542884 6448 ts_tablet_manager.cc:1939] T a546d54506a4494d91ff67c7edcd11cd P 0c32ebc1e12641008326a3653ffee651: Deleting consensus metadata
I20260810 14:43:12.543167 6445 ts_tablet_manager.cc:1918] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:43:12.543290 6445 log.cc:1196] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/a546d54506a4494d91ff67c7edcd11cd
I20260810 14:43:12.543598 6445 ts_tablet_manager.cc:1939] T a546d54506a4494d91ff67c7edcd11cd P c97ef34a1e344744aff36f73c4c2f5cc: Deleting consensus metadata
I20260810 14:43:12.544387 5643 catalog_manager.cc:4841] TS 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833): tablet a546d54506a4494d91ff67c7edcd11cd (table test-table [id=a331111502b9495ab66dc1f105951131]) successfully deleted
I20260810 14:43:12.544819 5643 catalog_manager.cc:4841] TS c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407): tablet a546d54506a4494d91ff67c7edcd11cd (table test-table [id=a331111502b9495ab66dc1f105951131]) successfully deleted
W20260810 14:43:12.547243 5647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:43507 (error 108)
I20260810 14:43:12.548091 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 5981
I20260810 14:43:12.548488 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:12.548083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43507 (local address 127.0.0.1:46294)
0810 14:43:12.548335 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:12.548403 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
W20260810 14:43:12.548651 5647 catalog_manager.cc:4584] TS c3275a4aca1f46e78bce20d3dee61493 (127.15.84.1:43507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a546d54506a4494d91ff67c7edcd11cd: Network error: Client connection negotiation failed: client connection to 127.15.84.1:43507: connect: Connection refused (error 111)
I20260810 14:43:12.565085 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 6184
I20260810 14:43:12.577804 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 5615
I20260810 14:43:12.591681 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:44379
--webserver_interface=127.15.84.62
--webserver_port=43541
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:44379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:12.709437 6452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:12.709757 6452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:12.709823 6452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:12.714061 6452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:43:12.714154 6452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:12.714180 6452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:12.714213 6452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:43:12.714241 6452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:43:12.719892 6452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:44379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:44379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=43541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:12.721046 6452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:12.722149 6452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:12.730772 6466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:12.730772 6462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:12.730818 6461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:12.731169 6452 server_base.cc:1034] running on GCE node
I20260810 14:43:12.731887 6452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:12.733111 6452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:12.734325 6452 hybrid_clock.cc:648] HybridClock initialized: now 1786372992734303 us; error 38 us; skew 500 ppm
Aug 10 14:43:12 dist-test-slave-rmhg krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372992, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:12.737330 6452 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:43:12.738384 6452 webserver.cc:460] Webserver started at http://127.15.84.62:43541/ using document root <none> and password file <none>
I20260810 14:43:12.738950 6452 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:12.739009 6452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:12.743106 6452 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:43:12.745426 6477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:12.746762 6452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:43:12.746883 6452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "ec3fabf5eb5640618d60ab5a672276c7"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "f9ef31186f69f891c8377339742cce63"
server_key_iv: "301eaef70eb47d512e47aad183ad37df"
server_key_version: "encryptionkey@0"
I20260810 14:43:12.747470 6452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:12.762850 6452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:12.765877 6452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:12.766070 6452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:12.773339 6452 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:44379
I20260810 14:43:12.773617 6566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:44379 every 8 connection(s)
I20260810 14:43:12.774523 6452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:43:12.778636 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 6452
I20260810 14:43:12.779083 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:43507
--local_ip_for_outbound_sockets=127.15.84.1
--tserver_master_addrs=127.15.84.62:44379
--webserver_port=35157
--webserver_interface=127.15.84.1
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:43:12.779110 6567 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:43:12.780925 6567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Bootstrap starting.
I20260810 14:43:12.799620 6567 log.cc:824] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:12.811396 6567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:43:12.811813 6567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7: Bootstrap complete.
I20260810 14:43:12.815960 6567 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } }
I20260810 14:43:12.816382 6567 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3fabf5eb5640618d60ab5a672276c7, State: Initialized, Role: FOLLOWER
I20260810 14:43:12.816886 6567 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } }
I20260810 14:43:12.817013 6567 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:43:12.817085 6567 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:43:12.817215 6567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 2 FOLLOWER]: Advancing to term 3
I20260810 14:43:12.819185 6567 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } }
I20260810 14:43:12.819525 6567 leader_election.cc:302] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec3fabf5eb5640618d60ab5a672276c7; no voters:
I20260810 14:43:12.819855 6567 leader_election.cc:288] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 14:43:12.819968 6573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 14:43:12.820179 6573 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [term 3 LEADER]: Becoming Leader. State: Replica: ec3fabf5eb5640618d60ab5a672276c7, State: Running, Role: LEADER
I20260810 14:43:12.820505 6573 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } }
I20260810 14:43:12.821210 6567 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:43:12.821969 6574 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ec3fabf5eb5640618d60ab5a672276c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } } }
I20260810 14:43:12.822116 6574 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:12.822643 6576 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec3fabf5eb5640618d60ab5a672276c7. Latest consensus state: current_term: 3 leader_uuid: "ec3fabf5eb5640618d60ab5a672276c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3fabf5eb5640618d60ab5a672276c7" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44379 } } }
I20260810 14:43:12.822799 6576 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3fabf5eb5640618d60ab5a672276c7 [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:12.823133 6585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:43:12.828994 6585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:43:12.829694 6585 catalog_manager.cc:1259] Loaded cluster ID: 2f5ecd296892408ebe15e06b3d075f71
I20260810 14:43:12.829759 6585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 14:43:12.831312 6585 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6ed608b9ad google::LogMessage::Fail() at ??:0
@ 0x7f6ed608e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6ed608b4de google::LogMessage::Flush() at ??:0
@ 0x7f6ed608ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6ee09cafdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6ee09cbd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6ee09c8579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6ee0a0c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6ed82dd3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6ed82ddc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6ed82df3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6ed82ce512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6ed99126db start_thread at ??:0
@ 0x7f6ed4b8961f clone at ??:0
W20260810 14:43:12.932217 6569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:12.932677 6569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:12.932747 6569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:12.939143 6569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:12.939251 6569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:12.939349 6569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:43:12.947856 6569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:43507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=35157
--tserver_master_addrs=127.15.84.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:12.949515 6569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:12.951040 6569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:12.964035 6640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:12.964071 6646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:12.964571 6569 server_base.cc:1034] running on GCE node
W20260810 14:43:12.964888 6642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:12.965488 6569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:12.966418 6569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:12.967633 6569 hybrid_clock.cc:648] HybridClock initialized: now 1786372992967595 us; error 55 us; skew 500 ppm
Aug 10 14:43:12 dist-test-slave-rmhg krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372992, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:12.979812 6569 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:43:12.981356 6569 webserver.cc:460] Webserver started at http://127.15.84.1:35157/ using document root <none> and password file <none>
I20260810 14:43:12.982187 6569 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:12.982311 6569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:12.988134 6569 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 14:43:12.991197 6655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:12.993005 6569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:43:12.993255 6569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "c3275a4aca1f46e78bce20d3dee61493"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "5a278a84b8b87c72ae96cda45c3937c9"
server_key_iv: "b3d71b8783c8e1b6db064f8836864b65"
server_key_version: "encryptionkey@0"
I20260810 14:43:12.994396 6569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:13.034777 6569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:13.040937 6569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:13.041247 6569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:13.042102 6569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:13.043728 6662 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 14:43:13.050164 6569 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 14:43:13.050279 6569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260810 14:43:13.050360 6569 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 14:43:13.053004 6569 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 14:43:13.053097 6569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260810 14:43:13.053634 6662 tablet_bootstrap.cc:492] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493: Bootstrap starting.
I20260810 14:43:13.076548 6569 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:43507
I20260810 14:43:13.076560 6846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:43507 every 8 connection(s)
I20260810 14:43:13.078037 6569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:43:13.078635 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 6569
I20260810 14:43:13.079136 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:33407
--local_ip_for_outbound_sockets=127.15.84.2
--tserver_master_addrs=127.15.84.62:44379
--webserver_port=37423
--webserver_interface=127.15.84.2
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:43:13.082935 6662 log.cc:824] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493: Log is configured to *not* fsync() on all Append() calls
W20260810 14:43:13.088140 6850 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:43:13.080550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:36967)
0810 14:43:13.083973 (+ 3423us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:13.084048 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:13.085300 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:13.086967 (+ 1667us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3152,"thread_start_us":180,"threads_started":1}
W20260810 14:43:13.089267 6847 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:44379 (0 consecutive failures): Network error: Failed to ping master at 127.15.84.62:44379: Client connection negotiation failed: client connection to 127.15.84.62:44379: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 14:43:13.090194 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:13.089900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:34555)
0810 14:43:13.090042 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:13.090118 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 14:43:13.091018 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:13.090812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:33019)
0810 14:43:13.090905 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:13.090956 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 14:43:13.091292 6847 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 14:43:13.097133 6662 tablet_bootstrap.cc:492] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:43:13.097797 6662 tablet_bootstrap.cc:492] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493: Bootstrap complete.
I20260810 14:43:13.099916 6662 ts_tablet_manager.cc:1397] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493: Time spent bootstrapping tablet: real 0.047s user 0.020s sys 0.016s
I20260810 14:43:13.103842 6662 raft_consensus.cc:348] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:13.104687 6662 raft_consensus.cc:729] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3275a4aca1f46e78bce20d3dee61493, State: Initialized, Role: FOLLOWER
I20260810 14:43:13.105445 6662 consensus_queue.cc:260] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:13.106979 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:13.106697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:58621)
0810 14:43:13.106828 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:13.106893 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 14:43:13.107748 6662 ts_tablet_manager.cc:1428] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 14:43:13.108589 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:13.108357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:35267)
0810 14:43:13.108467 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:13.108518 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 14:43:13.223358 6851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:13.223661 6851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:13.223724 6851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:13.227792 6851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:13.227875 6851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:13.227955 6851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:43:13.233553 6851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:33407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=37423
--tserver_master_addrs=127.15.84.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:13.234748 6851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:13.235805 6851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:13.246444 6865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:13.246686 6851 server_base.cc:1034] running on GCE node
W20260810 14:43:13.251019 6864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:13.251029 6868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:13.251655 6851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:13.252487 6851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:13.253675 6851 hybrid_clock.cc:648] HybridClock initialized: now 1786372993253653 us; error 50 us; skew 500 ppm
Aug 10 14:43:13 dist-test-slave-rmhg krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372993, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:13.256546 6851 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:43:13.257556 6851 webserver.cc:460] Webserver started at http://127.15.84.2:37423/ using document root <none> and password file <none>
I20260810 14:43:13.258117 6851 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:13.258183 6851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:13.262145 6851 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:43:13.264344 6879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:13.265686 6851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:43:13.265801 6851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "c97ef34a1e344744aff36f73c4c2f5cc"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "8c3b755b796082447971ceab470f93c2"
server_key_iv: "7ca93f023af9189b54a837b22281db2e"
server_key_version: "encryptionkey@0"
I20260810 14:43:13.266351 6851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:13.283689 6851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:13.286959 6851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:13.287122 6851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:13.287688 6851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:13.288520 6851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:13.288568 6851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:13.288609 6851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:13.288628 6851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:13.297305 6851 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:33407
I20260810 14:43:13.297364 7059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:33407 every 8 connection(s)
I20260810 14:43:13.298329 6851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:43:13.302538 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:13.300568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:39791)
0810 14:43:13.301050 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:13.301229 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":124,"threads_started":1}
W20260810 14:43:13.302996 7061 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:44379 (0 consecutive failures): Network error: Failed to ping master at 127.15.84.62:44379: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
I20260810 14:43:13.303606 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:13.303395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:33709)
0810 14:43:13.303497 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:13.303548 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 14:43:13.304157 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:13.303981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:51359)
0810 14:43:13.304063 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:13.304100 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 14:43:13.304317 7061 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 14:43:13.305004 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 6851
I20260810 14:43:13.305440 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:39833
--local_ip_for_outbound_sockets=127.15.84.3
--tserver_master_addrs=127.15.84.62:44379
--webserver_port=37283
--webserver_interface=127.15.84.3
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:13.422379 7065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:13.422695 7065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:13.422741 7065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:13.426776 7065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:13.426846 7065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:13.426903 7065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:43:13.432339 7065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:39833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=37283
--tserver_master_addrs=127.15.84.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:13.433475 7065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:13.434486 7065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:43:13.444082 7065 server_base.cc:1034] running on GCE node
W20260810 14:43:13.444029 7073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:13.444017 7072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:13.444022 7076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:13.444658 7065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:13.445349 7065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:13.446511 7065 hybrid_clock.cc:648] HybridClock initialized: now 1786372993446479 us; error 43 us; skew 500 ppm
Aug 10 14:43:13 dist-test-slave-rmhg krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372993, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:13.449218 7065 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:43:13.450095 7065 webserver.cc:460] Webserver started at http://127.15.84.3:37283/ using document root <none> and password file <none>
I20260810 14:43:13.450670 7065 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:13.450740 7065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:13.454766 7065 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:43:13.456862 7087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:13.458190 7065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:43:13.458312 7065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "0c32ebc1e12641008326a3653ffee651"
format_stamp: "Formatted at 2026-08-10 14:43:10 on dist-test-slave-rmhg"
server_key: "f80da2e6a5afb10db301ae9615f2f81d"
server_key_iv: "bb5ecffe6505035f7c518e5fcb6fded4"
server_key_version: "encryptionkey@0"
I20260810 14:43:13.458905 7065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:13.477864 7065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:13.481254 7065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:13.481463 7065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:13.482087 7065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:13.482973 7065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:13.483039 7065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:13.483088 7065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:13.483107 7065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:13.492316 7065 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:39833
I20260810 14:43:13.492345 7263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:39833 every 8 connection(s)
I20260810 14:43:13.493551 7065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:43:13.497684 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:13.495779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:57319)
0810 14:43:13.496288 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:13.496416 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":126,"threads_started":1}
W20260810 14:43:13.498150 7266 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:44379 (0 consecutive failures): Network error: Failed to ping master at 127.15.84.62:44379: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
I20260810 14:43:13.498746 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:13.498529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:58729)
0810 14:43:13.498630 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:13.498678 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 14:43:13.499315 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:13.499114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:44031)
0810 14:43:13.499218 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:13.499262 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 14:43:13.499485 7266 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 14:43:13.503536 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 7065
I20260810 14:43:14.110064 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:14.109463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:57543)
0810 14:43:14.109841 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:14.109925 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260810 14:43:14.305430 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:14.304946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:45425)
0810 14:43:14.305250 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:14.305314 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":80,"threads_started":1}
I20260810 14:43:14.500756 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:14.500220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:36695)
0810 14:43:14.500541 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:14.500606 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":96,"threads_started":1}
I20260810 14:43:14.795680 7279 raft_consensus.cc:482] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:43:14.795903 7279 raft_consensus.cc:504] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:14.797043 7279 leader_election.cc:288] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833), c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407)
Aug 10 14:43:14 dist-test-slave-rmhg krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372992, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.2@KRBTEST.COM
Aug 10 14:43:14 dist-test-slave-rmhg krb5kdc[4748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372992, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.3@KRBTEST.COM
I20260810 14:43:14.815212 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:14.797320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:39833 (local address 127.15.84.1:46505)
0810 14:43:14.797567 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:14.797583 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:14.797694 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:14.799479 (+ 1785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:14.799487 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:14.800167 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:14.800755 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:14.800774 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:14.802265 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:14.802268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:14.803286 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:14.803297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:14.804080 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:14.804084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:14.804329 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:14.808291 (+ 3962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:14.808322 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:14.810440 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:14.812621 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:14.812625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:14.812627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:14.812932 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:14.813336 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:14.813340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:14.813342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:14.813483 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:14.814157 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:14.814164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:14.814424 (+ 260us) client_negotiation.cc:769] Sending connection context
0810 14:43:14.814598 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 14:43:14.814982 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":162,"spinlock_wait_cycles":25472,"thread_start_us":75,"threads_started":1}
I20260810 14:43:14.815227 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:14.797466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:46505 (local address 127.15.84.3:39833)
0810 14:43:14.797755 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:14.797789 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:14.797835 (+ 46us) server_negotiation.cc:407] Connection header received
0810 14:43:14.798673 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:14.798681 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:14.799056 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:14.799334 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:14.800964 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:14.802098 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:14.803424 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:14.803946 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:14.804114 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:14.810682 (+ 6568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:14.810688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:14.810701 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:14.810734 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:14.812496 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:14.813055 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:14.813061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:14.813066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:14.813242 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:14.813573 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:14.813576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:14.813578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:14.814042 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:14.814169 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:14.814736 (+ 567us) server_negotiation.cc:299] Negotiation successful
0810 14:43:14.815059 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":82,"threads_started":1}
I20260810 14:43:14.815274 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:14.798047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:40421 (local address 127.15.84.2:33407)
0810 14:43:14.798381 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:14.798404 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:14.798437 (+ 33us) server_negotiation.cc:407] Connection header received
0810 14:43:14.799098 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:14.799104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:14.799495 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:14.799823 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:14.800948 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:14.802096 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:14.803428 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:14.803945 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:14.804121 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:14.810548 (+ 6427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:14.810555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:14.810571 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:14.810609 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:14.812410 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:14.813058 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:14.813064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:14.813068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:14.813261 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:14.813582 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:14.813585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:14.813587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:14.814043 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:14.814169 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:14.814753 (+ 584us) server_negotiation.cc:299] Negotiation successful
0810 14:43:14.815075 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":135,"threads_started":1}
I20260810 14:43:14.815266 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:14.797675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:33407 (local address 127.15.84.1:40421)
0810 14:43:14.797986 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:14.797998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:14.798087 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:14.799779 (+ 1692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:14.799782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:14.800157 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:14.800753 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:14.800772 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:14.802272 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:14.802276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:14.803280 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:14.803293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:14.804080 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:14.804083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:14.804333 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:14.808292 (+ 3959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:14.808318 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:14.810238 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:14.812553 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:14.812560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:14.812576 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:14.812935 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:14.813353 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:14.813356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:14.813358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:14.813491 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:14.814160 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:14.814167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:14.814424 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 14:43:14.814599 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 14:43:14.815074 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":303,"spinlock_wait_cycles":71936,"thread_start_us":58,"threads_started":1}
W20260810 14:43:14.818447 6656 leader_election.cc:341] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833): Not found: Tablet not found: a546d54506a4494d91ff67c7edcd11cd
W20260810 14:43:14.818723 6656 leader_election.cc:341] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407): Not found: Tablet not found: a546d54506a4494d91ff67c7edcd11cd
I20260810 14:43:14.818791 6656 leader_election.cc:302] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3275a4aca1f46e78bce20d3dee61493; no voters: 0c32ebc1e12641008326a3653ffee651, c97ef34a1e344744aff36f73c4c2f5cc
I20260810 14:43:14.819027 7279 raft_consensus.cc:2726] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:43:15.111272 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:15.110944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:53261)
0810 14:43:15.111100 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:15.111169 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 14:43:15.306710 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:15.306234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:49241)
0810 14:43:15.306533 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:15.306596 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":98,"threads_started":1}
I20260810 14:43:15.501986 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:15.501522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:42245)
0810 14:43:15.501820 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:15.501884 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
I20260810 14:43:16.112638 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:16.112090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:56557)
0810 14:43:16.112462 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:16.112531 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":83,"threads_started":1}
I20260810 14:43:16.308018 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:16.307560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:40799)
0810 14:43:16.307849 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:16.307915 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260810 14:43:16.503150 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:16.502716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:49129)
0810 14:43:16.502991 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:16.503051 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260810 14:43:16.632225 7294 raft_consensus.cc:482] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:43:16.632368 7294 raft_consensus.cc:504] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:16.632756 7294 leader_election.cc:288] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833), c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407)
W20260810 14:43:16.633662 6656 leader_election.cc:341] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833): Not found: Tablet not found: a546d54506a4494d91ff67c7edcd11cd
W20260810 14:43:16.633807 6656 leader_election.cc:341] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407): Not found: Tablet not found: a546d54506a4494d91ff67c7edcd11cd
I20260810 14:43:16.633858 6656 leader_election.cc:302] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3275a4aca1f46e78bce20d3dee61493; no voters: 0c32ebc1e12641008326a3653ffee651, c97ef34a1e344744aff36f73c4c2f5cc
I20260810 14:43:16.634025 7294 raft_consensus.cc:2726] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:43:17.113996 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:17.113550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:37047)
0810 14:43:17.113827 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:17.113891 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
I20260810 14:43:17.309387 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:17.308832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:58735)
0810 14:43:17.309186 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:17.309287 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":105,"threads_started":1}
I20260810 14:43:17.504451 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:17.503998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:36455)
0810 14:43:17.504282 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:17.504349 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":83,"threads_started":1}
I20260810 14:43:18.115401 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:18.114924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:51179)
0810 14:43:18.115230 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:18.115297 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":88,"threads_started":1}
I20260810 14:43:18.310788 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:18.310297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:45775)
0810 14:43:18.310596 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:18.310685 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":82,"threads_started":1}
I20260810 14:43:18.386765 7300 raft_consensus.cc:482] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:43:18.386911 7300 raft_consensus.cc:504] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:18.387362 7300 leader_election.cc:288] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833), c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407)
W20260810 14:43:18.388252 6656 leader_election.cc:341] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833): Not found: Tablet not found: a546d54506a4494d91ff67c7edcd11cd
W20260810 14:43:18.388386 6656 leader_election.cc:341] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407): Not found: Tablet not found: a546d54506a4494d91ff67c7edcd11cd
I20260810 14:43:18.388425 6656 leader_election.cc:302] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3275a4aca1f46e78bce20d3dee61493; no voters: 0c32ebc1e12641008326a3653ffee651, c97ef34a1e344744aff36f73c4c2f5cc
I20260810 14:43:18.388576 7300 raft_consensus.cc:2726] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:43:18.505826 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:18.505338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:46169)
0810 14:43:18.505659 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:18.505723 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":119,"threads_started":1}
I20260810 14:43:19.116842 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:19.116331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:54005)
0810 14:43:19.116672 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:19.116739 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
I20260810 14:43:19.312103 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:19.311636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:57163)
0810 14:43:19.311937 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:19.312001 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
I20260810 14:43:19.507032 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:19.506576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:40159)
0810 14:43:19.506865 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:19.506929 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":92,"threads_started":1}
I20260810 14:43:20.118337 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:20.117822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:35389)
0810 14:43:20.118156 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:20.118226 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
I20260810 14:43:20.252269 7310 raft_consensus.cc:482] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:43:20.252406 7310 raft_consensus.cc:504] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:20.252852 7310 leader_election.cc:288] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833), c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407)
W20260810 14:43:20.253770 6656 leader_election.cc:341] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833): Not found: Tablet not found: a546d54506a4494d91ff67c7edcd11cd
W20260810 14:43:20.253937 6656 leader_election.cc:341] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407): Not found: Tablet not found: a546d54506a4494d91ff67c7edcd11cd
I20260810 14:43:20.253995 6656 leader_election.cc:302] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3275a4aca1f46e78bce20d3dee61493; no voters: 0c32ebc1e12641008326a3653ffee651, c97ef34a1e344744aff36f73c4c2f5cc
I20260810 14:43:20.254163 7310 raft_consensus.cc:2726] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:43:20.313476 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:20.312880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:51781)
0810 14:43:20.313310 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:20.313375 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":116,"threads_started":1}
I20260810 14:43:20.508460 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:20.507928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:50397)
0810 14:43:20.508288 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:20.508356 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":83,"threads_started":1}
I20260810 14:43:21.119596 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:21.119151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:50705)
0810 14:43:21.119418 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:21.119481 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260810 14:43:21.314970 7317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:21.314416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:49863)
0810 14:43:21.314764 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:21.314843 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":125,"threads_started":1}
I20260810 14:43:21.509840 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:21.509380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:37629)
0810 14:43:21.509666 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:21.509736 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260810 14:43:22.120862 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:22.120414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:49751)
0810 14:43:22.120697 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:22.120756 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260810 14:43:22.316468 7320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:22.315879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:41113)
0810 14:43:22.316296 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:22.316366 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":109,"threads_started":1}
I20260810 14:43:22.511166 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:22.510700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:49973)
0810 14:43:22.510983 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:22.511052 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260810 14:43:23.122689 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:23.121856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:32997)
0810 14:43:23.122509 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:23.122570 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":86,"threads_started":1}
I20260810 14:43:23.317961 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:23.317470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:40435)
0810 14:43:23.317792 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:23.317860 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":83,"threads_started":1}
I20260810 14:43:23.512416 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:23.511976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:34303)
0810 14:43:23.512254 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:23.512314 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
I20260810 14:43:24.124125 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:24.123645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:48829)
0810 14:43:24.123944 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:24.124009 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260810 14:43:24.319299 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:24.318789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:51533)
0810 14:43:24.319128 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:24.319199 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":85,"threads_started":1}
I20260810 14:43:24.513737 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:24.513263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:48957)
0810 14:43:24.513541 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:24.513633 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260810 14:43:25.125584 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:25.125085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:42669)
0810 14:43:25.125416 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:25.125482 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":125,"threads_started":1}
I20260810 14:43:25.320686 7334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:25.320189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:46159)
0810 14:43:25.320511 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:25.320585 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":85,"threads_started":1}
I20260810 14:43:25.515203 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:25.514695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:34619)
0810 14:43:25.515011 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:25.515098 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":86,"threads_started":1}
I20260810 14:43:26.126854 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:26.126393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:46499)
0810 14:43:26.126691 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:26.126754 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":81,"threads_started":1}
I20260810 14:43:26.322144 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:26.321652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:36187)
0810 14:43:26.321965 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:26.322039 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":85,"threads_started":1}
I20260810 14:43:26.516566 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:26.516038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:52135)
0810 14:43:26.516390 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:26.516458 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":94,"threads_started":1}
I20260810 14:43:27.025942 7342 raft_consensus.cc:482] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:43:27.026099 7342 raft_consensus.cc:504] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c32ebc1e12641008326a3653ffee651" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39833 } } peers { permanent_uuid: "c97ef34a1e344744aff36f73c4c2f5cc" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 33407 } } peers { permanent_uuid: "c3275a4aca1f46e78bce20d3dee61493" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43507 } }
I20260810 14:43:27.026526 7342 leader_election.cc:288] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833), c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407)
W20260810 14:43:27.027351 6656 leader_election.cc:341] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c32ebc1e12641008326a3653ffee651 (127.15.84.3:39833): Not found: Tablet not found: a546d54506a4494d91ff67c7edcd11cd
W20260810 14:43:27.027514 6656 leader_election.cc:341] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c97ef34a1e344744aff36f73c4c2f5cc (127.15.84.2:33407): Not found: Tablet not found: a546d54506a4494d91ff67c7edcd11cd
I20260810 14:43:27.027565 6656 leader_election.cc:302] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3275a4aca1f46e78bce20d3dee61493; no voters: 0c32ebc1e12641008326a3653ffee651, c97ef34a1e344744aff36f73c4c2f5cc
I20260810 14:43:27.027763 7342 raft_consensus.cc:2726] T a546d54506a4494d91ff67c7edcd11cd P c3275a4aca1f46e78bce20d3dee61493 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:43:27.128187 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:27.127715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:37357)
0810 14:43:27.127996 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:27.128064 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260810 14:43:27.323590 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:27.323068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:59739)
0810 14:43:27.323419 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:27.323487 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":120,"threads_started":1}
I20260810 14:43:27.518193 7346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:27.517691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:52841)
0810 14:43:27.518016 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:27.518087 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":105,"threads_started":1}
I20260810 14:43:28.129808 7347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:28.129326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.1:55297)
0810 14:43:28.129627 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:28.129707 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260810 14:43:28.325188 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:28.324620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.2:57105)
0810 14:43:28.324953 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:28.325023 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":98,"threads_started":1}
I20260810 14:43:28.514358 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 6569
I20260810 14:43:28.520545 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:28.520126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44379 (local address 127.15.84.3:57637)
0810 14:43:28.520380 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:28.520440 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:44379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260810 14:43:28.527398 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 6851
I20260810 14:43:28.540930 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 7065
2026-08-10T14:43:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19753 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:43:28 dist-test-slave-rmhg krb5kdc[7354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:43:28 dist-test-slave-rmhg krb5kdc[7354](info): set up 2 sockets
Aug 10 14:43:28 dist-test-slave-rmhg krb5kdc[7354](info): commencing operation
krb5kdc: starting...
W20260810 14:43:29.698732 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:43:29 dist-test-slave-rmhg krb5kdc[7354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:43:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:43:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:43:29.830327 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40369
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:40369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:29.947109 7382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:29.947402 7382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:29.947443 7382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:29.951603 7382 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 14:43:29.951669 7382 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 14:43:29.951684 7382 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 14:43:29.951696 7382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:43:29.951713 7382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:29.951730 7382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:29.951749 7382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:43:29.951779 7382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:43:29.957291 7382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:40369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:29.958433 7382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:29.959497 7382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:29.968305 7394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:29.968371 7393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:29.968309 7397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:29.968364 7382 server_base.cc:1034] running on GCE node
I20260810 14:43:29.969134 7382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:29.970340 7382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:29.971503 7382 hybrid_clock.cc:648] HybridClock initialized: now 1786373009971466 us; error 50 us; skew 500 ppm
Aug 10 14:43:29 dist-test-slave-rmhg krb5kdc[7354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373009, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:29.974293 7382 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:43:29.975217 7382 webserver.cc:460] Webserver started at http://127.15.84.62:37561/ using document root <none> and password file <none>
I20260810 14:43:29.975760 7382 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:29.975826 7382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:29.976023 7382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:43:29.978147 7382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "9df69aa180b24b8eb01b4d41af759d0f"
format_stamp: "Formatted at 2026-08-10 14:43:29 on dist-test-slave-rmhg"
server_key: "8f5d6b9a3390c09984c0ad9f8957cb50"
server_key_iv: "4ab92e5d49d95c4730a983f3415c1e2a"
server_key_version: "encryptionkey@0"
I20260810 14:43:29.978690 7382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "9df69aa180b24b8eb01b4d41af759d0f"
format_stamp: "Formatted at 2026-08-10 14:43:29 on dist-test-slave-rmhg"
server_key: "8f5d6b9a3390c09984c0ad9f8957cb50"
server_key_iv: "4ab92e5d49d95c4730a983f3415c1e2a"
server_key_version: "encryptionkey@0"
I20260810 14:43:29.982466 7382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:43:29.984871 7409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:29.986110 7382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:43:29.986246 7382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "9df69aa180b24b8eb01b4d41af759d0f"
format_stamp: "Formatted at 2026-08-10 14:43:29 on dist-test-slave-rmhg"
server_key: "8f5d6b9a3390c09984c0ad9f8957cb50"
server_key_iv: "4ab92e5d49d95c4730a983f3415c1e2a"
server_key_version: "encryptionkey@0"
I20260810 14:43:29.986346 7382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:30.009064 7382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:30.012483 7382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:30.012678 7382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:30.019842 7382 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:40369
I20260810 14:43:30.020113 7489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:40369 every 8 connection(s)
I20260810 14:43:30.020874 7382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:43:30.023761 7492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:30.028574 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 7382
I20260810 14:43:30.028721 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:43:30.028983 15696 external_mini_cluster.cc:1428] Setting key a57741b019baeab3aeea87b5a37de17a
I20260810 14:43:30.030282 7492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Bootstrap starting.
I20260810 14:43:30.032733 7492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:30.033696 7492 log.cc:824] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:30.035977 7492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: No bootstrap required, opened a new log
Aug 10 14:43:30 dist-test-slave-rmhg krb5kdc[7354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:30.040099 7492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } }
I20260810 14:43:30.040328 7492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:30.040367 7492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df69aa180b24b8eb01b4d41af759d0f, State: Initialized, Role: FOLLOWER
I20260810 14:43:30.040817 7492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } }
I20260810 14:43:30.040940 7492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:43:30.041044 7492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:43:30.041172 7492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:30.042063 7492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } }
I20260810 14:43:30.042394 7492 leader_election.cc:302] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9df69aa180b24b8eb01b4d41af759d0f; no voters:
I20260810 14:43:30.042724 7492 leader_election.cc:288] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:43:30.042830 7501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:43:30.043116 7501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 1 LEADER]: Becoming Leader. State: Replica: 9df69aa180b24b8eb01b4d41af759d0f, State: Running, Role: LEADER
I20260810 14:43:30.043489 7501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } }
I20260810 14:43:30.044064 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:30.030580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.15.84.62:40369)
0810 14:43:30.031034 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:30.031052 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:30.031088 (+ 36us) server_negotiation.cc:407] Connection header received
0810 14:43:30.031766 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:30.031786 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:30.032145 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:30.032436 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:30.033409 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.034678 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.035818 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.036338 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.036455 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:30.039171 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:30.039176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:30.039190 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:30.039218 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:30.040898 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:30.041389 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:30.041394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:30.041399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:30.041515 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:30.041786 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:30.041789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:30.041790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:30.042200 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:30.042433 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:30.042752 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 14:43:30.043086 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":123,"threads_started":1}
I20260810 14:43:30.044129 7492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:43:30.044695 7501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9df69aa180b24b8eb01b4d41af759d0f. Latest consensus state: current_term: 1 leader_uuid: "9df69aa180b24b8eb01b4d41af759d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } } }
I20260810 14:43:30.044828 7501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:30.045466 7514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:43:30.048640 7514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:43:30.048456 7503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9df69aa180b24b8eb01b4d41af759d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } } }
I20260810 14:43:30.048967 7503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:30.054345 7514 catalog_manager.cc:1347] Generated new cluster ID: 44ebccd549674e85bfa510a806d6d444
I20260810 14:43:30.054417 7514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:43:30.079919 7514 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:43:30.081017 7514 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:43:30.088016 7514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Generated new TSK 0
I20260810 14:43:30.088752 7514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:43:30.124264 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40369
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:30.240430 7540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:30.240729 7540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:30.240792 7540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:30.244993 7540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:30.245085 7540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:30.245193 7540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:43:30.250733 7540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:30.251919 7540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:30.252969 7540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:30.262796 7551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:30.263032 7540 server_base.cc:1034] running on GCE node
W20260810 14:43:30.262822 7553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:30.262877 7550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:30.263635 7540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:30.264374 7540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:30.265538 7540 hybrid_clock.cc:648] HybridClock initialized: now 1786373010265505 us; error 54 us; skew 500 ppm
Aug 10 14:43:30 dist-test-slave-rmhg krb5kdc[7354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373010, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:30.268313 7540 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:43:30.269294 7540 webserver.cc:460] Webserver started at http://127.15.84.1:34139/ using document root <none> and password file <none>
I20260810 14:43:30.269863 7540 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:30.269933 7540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:30.270152 7540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:43:30.272259 7540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "9b8b2e084cec486c8212030926e3e01e"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "b0cfd9195d069754b11749d11afcc624"
server_key_iv: "fd4a8c3cd407b068b67c559f8eaf63d5"
server_key_version: "encryptionkey@0"
I20260810 14:43:30.272809 7540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "9b8b2e084cec486c8212030926e3e01e"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "b0cfd9195d069754b11749d11afcc624"
server_key_iv: "fd4a8c3cd407b068b67c559f8eaf63d5"
server_key_version: "encryptionkey@0"
I20260810 14:43:30.276798 7540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 14:43:30.279438 7564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:30.280742 7540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:43:30.280864 7540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "9b8b2e084cec486c8212030926e3e01e"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "b0cfd9195d069754b11749d11afcc624"
server_key_iv: "fd4a8c3cd407b068b67c559f8eaf63d5"
server_key_version: "encryptionkey@0"
I20260810 14:43:30.280973 7540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:30.290562 7540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:30.293332 7540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:30.293503 7540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:30.294078 7540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:30.295048 7540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:30.295114 7540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:30.295162 7540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:30.295187 7540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:30.303989 7540 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:43907
I20260810 14:43:30.304014 7761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:43907 every 8 connection(s)
I20260810 14:43:30.304970 7540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:43:30.312808 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 7540
I20260810 14:43:30.312949 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:43:30.313277 15696 external_mini_cluster.cc:1428] Setting key 9ae5f333772cbd7e9b3d63fb30d6ec0e
Aug 10 14:43:30 dist-test-slave-rmhg krb5kdc[7354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373010, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:30.319729 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:30.306862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:46317 (local address 127.15.84.62:40369)
0810 14:43:30.307030 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:30.307035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:30.307949 (+ 914us) server_negotiation.cc:407] Connection header received
0810 14:43:30.309360 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:30.309364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:30.309418 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:30.309499 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:30.310796 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.311371 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.312582 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.312833 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.312949 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:30.315941 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:30.315946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:30.315948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:30.315981 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:30.317454 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:30.318086 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:30.318089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:30.318091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:30.318180 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:30.318548 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:30.318551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:30.318553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:30.318781 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:30.318875 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:30.319425 (+ 550us) server_negotiation.cc:299] Negotiation successful
0810 14:43:30.319566 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:43:30.320571 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:30.307288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:46317)
0810 14:43:30.307788 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:30.307833 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:30.309094 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:30.309700 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:30.309708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:30.310094 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:30.310607 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:30.310622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.311482 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.311484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:30.312439 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:30.312449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.313041 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.313046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:30.313296 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:30.313926 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:30.313944 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:30.315751 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:30.317573 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:30.317580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:30.317600 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:30.317977 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:30.318269 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:30.318272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:30.318274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:30.318466 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:30.318879 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:30.318885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:30.319178 (+ 293us) client_negotiation.cc:769] Sending connection context
0810 14:43:30.319349 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 14:43:30.319651 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":130,"threads_started":1}
I20260810 14:43:30.321990 7765 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40369
I20260810 14:43:30.322258 7765 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:30.322804 7765 heartbeater.cc:511] Master 127.15.84.62:40369 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:43:30.324609 7432 ts_manager.cc:194] Registered new tserver with Master: 9b8b2e084cec486c8212030926e3e01e (127.15.84.1:43907)
I20260810 14:43:30.325937 7432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:46317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:43:30.344220 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40369
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:30.465176 7772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:30.465492 7772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:30.465545 7772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:30.469883 7772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:30.469974 7772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:30.470046 7772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:43:30.475873 7772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:30.477108 7772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:30.478261 7772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:30.488960 7780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:30.489193 7772 server_base.cc:1034] running on GCE node
W20260810 14:43:30.488996 7786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:30.488962 7781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:30.489926 7772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:30.490666 7772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:30.491848 7772 hybrid_clock.cc:648] HybridClock initialized: now 1786373010491822 us; error 35 us; skew 500 ppm
Aug 10 14:43:30 dist-test-slave-rmhg krb5kdc[7354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373010, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:30.494822 7772 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:43:30.496063 7772 webserver.cc:460] Webserver started at http://127.15.84.2:37865/ using document root <none> and password file <none>
I20260810 14:43:30.496891 7772 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:30.496989 7772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:30.497289 7772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:43:30.500176 7772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "fb5125fb1b9941cc99710ee7bd6282f6"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "503a360583b1c291a3e05693f1f74606"
server_key_iv: "623010ce544cd964a6a19502ece517f1"
server_key_version: "encryptionkey@0"
I20260810 14:43:30.500919 7772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "fb5125fb1b9941cc99710ee7bd6282f6"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "503a360583b1c291a3e05693f1f74606"
server_key_iv: "623010ce544cd964a6a19502ece517f1"
server_key_version: "encryptionkey@0"
I20260810 14:43:30.508452 7772 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260810 14:43:30.511446 7795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:30.512923 7772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:43:30.513077 7772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "fb5125fb1b9941cc99710ee7bd6282f6"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "503a360583b1c291a3e05693f1f74606"
server_key_iv: "623010ce544cd964a6a19502ece517f1"
server_key_version: "encryptionkey@0"
I20260810 14:43:30.513248 7772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:30.548070 7772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:30.551373 7772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:30.551574 7772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:30.552224 7772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:30.553190 7772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:30.553249 7772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:30.553318 7772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:30.553341 7772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:30.562690 7772 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:38591
I20260810 14:43:30.562762 7975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:38591 every 8 connection(s)
I20260810 14:43:30.563697 7772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:43:30.564824 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 7772
I20260810 14:43:30.564951 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:43:30.565300 15696 external_mini_cluster.cc:1428] Setting key 7a101c2fa99be8bb89ca7cb9dbdd6c2c
Aug 10 14:43:30 dist-test-slave-rmhg krb5kdc[7354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373010, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
I20260810 14:43:30.579288 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:30.565873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:57589 (local address 127.15.84.62:40369)
0810 14:43:30.566029 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:30.566034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:30.567155 (+ 1121us) server_negotiation.cc:407] Connection header received
0810 14:43:30.568623 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:30.568627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:30.568687 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:30.568765 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:30.570241 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.570784 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.572029 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.572280 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.572377 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:30.575090 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:30.575094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:30.575096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:30.575126 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:30.576777 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:30.577461 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:30.577465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:30.577466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:30.577561 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:30.577940 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:30.577944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:30.577945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:30.578152 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:30.578244 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:30.578969 (+ 725us) server_negotiation.cc:299] Negotiation successful
0810 14:43:30.579117 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:43:30.580273 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:30.566398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:57589)
0810 14:43:30.566957 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:30.567017 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:30.568346 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:30.568993 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:30.569004 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:30.569524 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:30.570077 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:30.570095 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.570922 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.570926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:30.571892 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:30.571904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.572447 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.572452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:30.572651 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:30.573281 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:30.573300 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:30.574873 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:30.576912 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:30.576920 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:30.576940 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:30.577349 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:30.577658 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:30.577661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:30.577663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:30.577834 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:30.578263 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:30.578271 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:30.578657 (+ 386us) client_negotiation.cc:769] Sending connection context
0810 14:43:30.578878 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 14:43:30.579270 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":136,"threads_started":1}
I20260810 14:43:30.581599 7977 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40369
I20260810 14:43:30.581874 7977 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:30.582422 7977 heartbeater.cc:511] Master 127.15.84.62:40369 requested a full tablet report, sending...
I20260810 14:43:30.583668 7432 ts_manager.cc:194] Registered new tserver with Master: fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591)
I20260810 14:43:30.584483 7432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:57589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:43:30.596298 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40369
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:30.714929 7987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:30.715238 7987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:30.715283 7987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:30.719578 7987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:30.719666 7987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:30.719726 7987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:43:30.725428 7987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:30.726591 7987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:30.727713 7987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:30.738133 7995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:30.738292 7987 server_base.cc:1034] running on GCE node
W20260810 14:43:30.738143 7993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:30.738160 7997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:30.738875 7987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:30.739548 7987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:30.740718 7987 hybrid_clock.cc:648] HybridClock initialized: now 1786373010740698 us; error 32 us; skew 500 ppm
Aug 10 14:43:30 dist-test-slave-rmhg krb5kdc[7354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373010, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:30.743620 7987 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:43:30.744634 7987 webserver.cc:460] Webserver started at http://127.15.84.3:44875/ using document root <none> and password file <none>
I20260810 14:43:30.745263 7987 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:30.745333 7987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:30.745553 7987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:43:30.747737 7987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "f9dd5ae1fae6450a8ce49b16637ddde7"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "641117d730c0211ba3f481b2fca4e27a"
server_key_iv: "96694a86fd0e30235ae972fa76e5a47c"
server_key_version: "encryptionkey@0"
I20260810 14:43:30.748272 7987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "f9dd5ae1fae6450a8ce49b16637ddde7"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "641117d730c0211ba3f481b2fca4e27a"
server_key_iv: "96694a86fd0e30235ae972fa76e5a47c"
server_key_version: "encryptionkey@0"
I20260810 14:43:30.752153 7987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:43:30.754861 8012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:30.756217 7987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 14:43:30.756340 7987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "f9dd5ae1fae6450a8ce49b16637ddde7"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "641117d730c0211ba3f481b2fca4e27a"
server_key_iv: "96694a86fd0e30235ae972fa76e5a47c"
server_key_version: "encryptionkey@0"
I20260810 14:43:30.756467 7987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:30.777889 7987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:30.781629 7987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:30.781834 7987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:30.782539 7987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:30.783461 7987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:30.783532 7987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:30.783591 7987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:30.783617 7987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:30.793386 7987 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:40217
I20260810 14:43:30.793409 8194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:40217 every 8 connection(s)
I20260810 14:43:30.794405 7987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
Aug 10 14:43:30 dist-test-slave-rmhg krb5kdc[7354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373010, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:30.804688 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 7987
I20260810 14:43:30.804821 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:43:30.805097 15696 external_mini_cluster.cc:1428] Setting key 4e3b3dfd1aea0b3189deab98d68ec850
I20260810 14:43:30.808650 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:30.796234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:50061 (local address 127.15.84.62:40369)
0810 14:43:30.796368 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:30.796372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:30.797342 (+ 970us) server_negotiation.cc:407] Connection header received
0810 14:43:30.798754 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:30.798758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:30.798820 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:30.798899 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:30.800204 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.800749 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.802010 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.802240 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.802336 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:30.804822 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:30.804826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:30.804829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:30.804861 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:30.806586 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:30.807140 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:30.807144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:30.807146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:30.807241 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:30.807629 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:30.807632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:30.807633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:30.807831 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:30.807906 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:30.808412 (+ 506us) server_negotiation.cc:299] Negotiation successful
0810 14:43:30.808525 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:43:30.809554 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:30.796598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:50061)
0810 14:43:30.797136 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:30.797227 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:30.798515 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:30.799115 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:30.799123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:30.799505 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:30.800046 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:30.800060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.800880 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.800883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:30.801876 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:30.801887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.802396 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.802400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:30.802573 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:30.803126 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:30.803143 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:30.804635 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:30.806709 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:30.806717 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:30.806735 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:30.807040 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:30.807341 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:30.807344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:30.807345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:30.807543 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:30.807908 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:30.807913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:30.808184 (+ 271us) client_negotiation.cc:769] Sending connection context
0810 14:43:30.808351 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 14:43:30.808632 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":126,"threads_started":1}
I20260810 14:43:30.810773 8197 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40369
I20260810 14:43:30.811034 8197 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:30.811550 8197 heartbeater.cc:511] Master 127.15.84.62:40369 requested a full tablet report, sending...
I20260810 14:43:30.812656 7432 ts_manager.cc:194] Registered new tserver with Master: f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217)
I20260810 14:43:30.813280 7432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:50061
I20260810 14:43:30.819355 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:43:30.828449 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:30.821446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.15.84.62:40369)
0810 14:43:30.821593 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:30.821597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:30.821742 (+ 145us) server_negotiation.cc:407] Connection header received
0810 14:43:30.821848 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:30.821850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:30.821894 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:30.821960 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:30.822609 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.823127 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.823914 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.824141 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.824225 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:30.825627 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:30.825630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:30.825632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:30.825659 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:30.826929 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:30.827357 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:30.827360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:30.827361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:30.827454 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:30.827657 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:30.827659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:30.827660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:30.827893 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:30.827996 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:30.828172 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 14:43:30.828284 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:43:30.831537 7432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:43:30.834913 7432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:43:30.850075 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:30.844162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44686 (local address 127.15.84.1:43907)
0810 14:43:30.844687 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:30.844694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:30.845092 (+ 398us) server_negotiation.cc:407] Connection header received
0810 14:43:30.845354 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:30.845360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:30.845570 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:30.845731 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:30.845938 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.846869 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.848627 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.849336 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.849450 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:30.849623 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:30.849806 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 14:43:30.849916 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":132,"threads_started":1}
I20260810 14:43:30.850065 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:30.843719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43907 (local address 127.0.0.1:44686)
0810 14:43:30.844998 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:30.845010 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:30.845204 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:30.845709 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:30.845711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:30.845730 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:30.845823 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:30.845829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.846976 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.846979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:30.848485 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:30.848496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.849470 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.849474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:30.849586 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:30.849626 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 14:43:30.849766 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 14:43:30.849899 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"mutex_wait_us":69,"thread_start_us":73,"threads_started":1}
I20260810 14:43:30.850526 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:30.844478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:38591 (local address 127.0.0.1:59702)
0810 14:43:30.845199 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:30.845213 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:30.845299 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:30.846130 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:30.846145 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:30.846185 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:30.846290 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:30.846297 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.847712 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.847715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:30.848691 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:30.848703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.849796 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.849799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:30.849891 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:30.849913 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:43:30.849980 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 14:43:30.850037 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":50,"thread_start_us":87,"threads_started":1}
I20260810 14:43:30.850673 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:30.843706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:40217 (local address 127.0.0.1:58842)
0810 14:43:30.844594 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:30.844649 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:30.844779 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:30.845454 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:30.845460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:30.845481 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:30.845630 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:30.845636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.847186 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.847189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:30.848485 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:30.848496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.850274 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.850277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:30.850354 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:30.850380 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:43:30.850433 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 14:43:30.850502 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":61,"thread_start_us":131,"threads_started":1}
I20260810 14:43:30.851050 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:30.844739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.15.84.2:38591)
0810 14:43:30.845516 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:30.845528 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:30.845550 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:43:30.845624 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:30.845629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:30.845806 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:30.845947 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:30.846418 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.847563 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.848810 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.849472 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.849580 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:30.849735 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:30.850495 (+ 760us) server_negotiation.cc:299] Negotiation successful
0810 14:43:30.850920 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":124,"threads_started":1}
I20260810 14:43:30.851397 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:30.843891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58842 (local address 127.15.84.3:40217)
0810 14:43:30.844884 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:30.844895 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:30.844921 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:43:30.845003 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:30.845008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:30.845279 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:30.845795 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:30.845961 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.847052 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.849279 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:30.850048 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:30.850929 (+ 881us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:30.851096 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:30.851154 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 14:43:30.851248 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":161,"threads_started":1}
I20260810 14:43:30.853662 7647 tablet_service.cc:1467] Processing CreateTablet for tablet 7246153b9f134f81878b945aeb501d25 (DEFAULT_TABLE table=test-table [id=0a2d6854d5a147028c70843fcd2f9e6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:30.854286 8082 tablet_service.cc:1467] Processing CreateTablet for tablet 7246153b9f134f81878b945aeb501d25 (DEFAULT_TABLE table=test-table [id=0a2d6854d5a147028c70843fcd2f9e6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:30.854939 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7246153b9f134f81878b945aeb501d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:30.855379 8082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7246153b9f134f81878b945aeb501d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:30.856832 7870 tablet_service.cc:1467] Processing CreateTablet for tablet 7246153b9f134f81878b945aeb501d25 (DEFAULT_TABLE table=test-table [id=0a2d6854d5a147028c70843fcd2f9e6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:30.857741 7870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7246153b9f134f81878b945aeb501d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:30.862711 8226 tablet_bootstrap.cc:492] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7: Bootstrap starting.
I20260810 14:43:30.863216 8228 tablet_bootstrap.cc:492] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Bootstrap starting.
I20260810 14:43:30.865171 8230 tablet_bootstrap.cc:492] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6: Bootstrap starting.
I20260810 14:43:30.865833 8228 tablet_bootstrap.cc:654] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:30.866706 8228 log.cc:824] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:30.867848 8230 tablet_bootstrap.cc:654] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:30.868731 8230 log.cc:824] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:30.868924 8226 tablet_bootstrap.cc:654] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:30.869002 8228 tablet_bootstrap.cc:492] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: No bootstrap required, opened a new log
I20260810 14:43:30.869251 8228 ts_tablet_manager.cc:1397] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 14:43:30.870151 8226 log.cc:824] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:30.870983 8230 tablet_bootstrap.cc:492] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6: No bootstrap required, opened a new log
I20260810 14:43:30.871219 8230 ts_tablet_manager.cc:1397] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:43:30.871891 8226 tablet_bootstrap.cc:492] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7: No bootstrap required, opened a new log
I20260810 14:43:30.872077 8226 ts_tablet_manager.cc:1397] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20260810 14:43:30.873911 8228 raft_consensus.cc:348] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } }
I20260810 14:43:30.874210 8228 raft_consensus.cc:374] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:30.874262 8228 raft_consensus.cc:729] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8b2e084cec486c8212030926e3e01e, State: Initialized, Role: FOLLOWER
I20260810 14:43:30.874821 8228 consensus_queue.cc:260] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } }
I20260810 14:43:30.875097 8230 raft_consensus.cc:348] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } }
I20260810 14:43:30.875394 8230 raft_consensus.cc:374] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:30.875453 8230 raft_consensus.cc:729] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5125fb1b9941cc99710ee7bd6282f6, State: Initialized, Role: FOLLOWER
I20260810 14:43:30.875378 8226 raft_consensus.cc:348] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } }
I20260810 14:43:30.875658 8226 raft_consensus.cc:374] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:30.875710 8226 raft_consensus.cc:729] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9dd5ae1fae6450a8ce49b16637ddde7, State: Initialized, Role: FOLLOWER
I20260810 14:43:30.875764 7765 heartbeater.cc:503] Master 127.15.84.62:40369 was elected leader, sending a full tablet report...
I20260810 14:43:30.875979 8228 ts_tablet_manager.cc:1428] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 14:43:30.875942 8230 consensus_queue.cc:260] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } }
I20260810 14:43:30.876185 8226 consensus_queue.cc:260] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } }
I20260810 14:43:30.876696 7977 heartbeater.cc:503] Master 127.15.84.62:40369 was elected leader, sending a full tablet report...
I20260810 14:43:30.877169 8226 ts_tablet_manager.cc:1428] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260810 14:43:30.877489 8197 heartbeater.cc:503] Master 127.15.84.62:40369 was elected leader, sending a full tablet report...
I20260810 14:43:30.877861 8230 ts_tablet_manager.cc:1428] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260810 14:43:31.045498 8199 tablet.cc:2369] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:43:31.056499 7766 tablet.cc:2369] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:43:31.064944 7980 tablet.cc:2369] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:43:31.122833 8236 raft_consensus.cc:482] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:43:31.123086 8236 raft_consensus.cc:504] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } }
I20260810 14:43:31.124364 8236 leader_election.cc:288] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217), fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591)
I20260810 14:43:31.129804 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:31.124753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:34729 (local address 127.15.84.3:40217)
0810 14:43:31.124902 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:31.124906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:31.125238 (+ 332us) server_negotiation.cc:407] Connection header received
0810 14:43:31.125381 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:31.125384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:31.125438 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:31.125563 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:31.125927 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:31.126745 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:31.128622 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:31.129248 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:31.129341 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:31.129405 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:31.129613 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 14:43:31.129675 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:43:31.129933 8240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:31.124675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:38591 (local address 127.15.84.1:51073)
0810 14:43:31.125134 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:31.125145 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:31.125270 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:31.125542 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:31.125545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:31.125562 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:31.125778 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:31.125784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:31.126898 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:31.126902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:31.128559 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:31.128569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:31.129566 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:31.129569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:31.129675 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:31.129695 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 14:43:31.129762 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 14:43:31.129816 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":88,"spinlock_wait_cycles":59392,"thread_start_us":113,"threads_started":1}
I20260810 14:43:31.129997 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:31.124817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:51073 (local address 127.15.84.2:38591)
0810 14:43:31.124955 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:31.124959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:31.125256 (+ 297us) server_negotiation.cc:407] Connection header received
0810 14:43:31.125380 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:31.125383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:31.125438 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:31.125526 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:31.125927 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:31.126745 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:31.128692 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:31.129331 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:31.129414 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:31.129479 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:31.129803 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 14:43:31.129886 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:43:31.129854 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:31.124673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:40217 (local address 127.15.84.1:34729)
0810 14:43:31.125080 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:31.125096 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:31.125271 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:31.125570 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:31.125572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:31.125585 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:31.125768 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:31.125777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:31.126898 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:31.126902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:31.128418 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:31.128432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:31.129381 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:31.129385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:31.129485 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:31.129523 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 14:43:31.129600 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:43:31.129683 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":220,"thread_start_us":146,"threads_started":1}
I20260810 14:43:31.130642 7902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7246153b9f134f81878b945aeb501d25" candidate_uuid: "9b8b2e084cec486c8212030926e3e01e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" is_pre_election: true
I20260810 14:43:31.130962 8114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7246153b9f134f81878b945aeb501d25" candidate_uuid: "9b8b2e084cec486c8212030926e3e01e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" is_pre_election: true
I20260810 14:43:31.131006 7902 raft_consensus.cc:2454] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b8b2e084cec486c8212030926e3e01e in term 0.
I20260810 14:43:31.131240 8114 raft_consensus.cc:2454] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b8b2e084cec486c8212030926e3e01e in term 0.
I20260810 14:43:31.131505 7570 leader_election.cc:302] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b8b2e084cec486c8212030926e3e01e, fb5125fb1b9941cc99710ee7bd6282f6; no voters:
I20260810 14:43:31.131835 8236 raft_consensus.cc:2781] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:43:31.131930 8236 raft_consensus.cc:482] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:43:31.131980 8236 raft_consensus.cc:3037] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:31.133056 8236 raft_consensus.cc:504] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } }
I20260810 14:43:31.133440 8236 leader_election.cc:288] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 1 election: Requested vote from peers f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217), fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591)
I20260810 14:43:31.133807 8114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7246153b9f134f81878b945aeb501d25" candidate_uuid: "9b8b2e084cec486c8212030926e3e01e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7"
I20260810 14:43:31.133816 7902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7246153b9f134f81878b945aeb501d25" candidate_uuid: "9b8b2e084cec486c8212030926e3e01e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5125fb1b9941cc99710ee7bd6282f6"
I20260810 14:43:31.133932 7902 raft_consensus.cc:3037] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:31.133939 8114 raft_consensus.cc:3037] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:31.135051 7902 raft_consensus.cc:2454] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b8b2e084cec486c8212030926e3e01e in term 1.
I20260810 14:43:31.135182 8114 raft_consensus.cc:2454] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b8b2e084cec486c8212030926e3e01e in term 1.
I20260810 14:43:31.135399 7570 leader_election.cc:302] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b8b2e084cec486c8212030926e3e01e, fb5125fb1b9941cc99710ee7bd6282f6; no voters:
I20260810 14:43:31.135634 8236 raft_consensus.cc:2781] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:43:31.135885 8236 raft_consensus.cc:686] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 1 LEADER]: Becoming Leader. State: Replica: 9b8b2e084cec486c8212030926e3e01e, State: Running, Role: LEADER
I20260810 14:43:31.136304 8236 consensus_queue.cc:237] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } }
I20260810 14:43:31.139863 7430 catalog_manager.cc:5495] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b8b2e084cec486c8212030926e3e01e (127.15.84.1). New cstate: current_term: 1 leader_uuid: "9b8b2e084cec486c8212030926e3e01e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } health_report { overall_health: UNKNOWN } } }
I20260810 14:43:31.216610 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:31.212828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.15.84.1:43907)
0810 14:43:31.212982 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:31.212986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:31.213003 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:43:31.213191 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:31.213193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:31.213261 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:31.213360 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:43:31.213644 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:31.214162 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:31.215065 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:31.215305 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:31.215399 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:31.215739 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:43:31.216198 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:43:31.216313 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:31.216385 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 14:43:31.216451 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:43:31.223122 8114 raft_consensus.cc:1261] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 1 FOLLOWER]: Refusing update from remote peer 9b8b2e084cec486c8212030926e3e01e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:43:31.223129 7902 raft_consensus.cc:1261] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6 [term 1 FOLLOWER]: Refusing update from remote peer 9b8b2e084cec486c8212030926e3e01e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:43:31.223896 8244 consensus_queue.cc:1035] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:43:31.224098 8236 consensus_queue.cc:1035] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:43:31.232654 8250 mvcc.cc:205] Tried to move back new op lower bound from 7316983853962080256 to 7316983853616979968. Current Snapshot: MvccSnapshot[applied={T|T < 7316983853962080256}]
I20260810 14:43:31.241437 7430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:43:31.241657 7430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:43:31.244376 7430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Sending DeleteTablet for 3 replicas of tablet 7246153b9f134f81878b945aeb501d25
I20260810 14:43:31.245302 7647 tablet_service.cc:1514] Processing DeleteTablet for tablet 7246153b9f134f81878b945aeb501d25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:31 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:44686
I20260810 14:43:31.245297 8082 tablet_service.cc:1514] Processing DeleteTablet for tablet 7246153b9f134f81878b945aeb501d25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:31 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:58842
I20260810 14:43:31.245393 7870 tablet_service.cc:1514] Processing DeleteTablet for tablet 7246153b9f134f81878b945aeb501d25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:31 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:59702
I20260810 14:43:31.245980 8264 tablet_replica.cc:330] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7: stopping tablet replica
I20260810 14:43:31.246071 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 7540
I20260810 14:43:31.253548 8262 tablet_replica.cc:330] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6: stopping tablet replica
I20260810 14:43:31.253530 8264 raft_consensus.cc:2229] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:43:31.253899 8262 raft_consensus.cc:2229] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:43:31.253963 8264 raft_consensus.cc:2258] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:43:31.254544 8262 raft_consensus.cc:2258] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:43:31.255795 8264 ts_tablet_manager.cc:1905] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:43:31.255961 8262 ts_tablet_manager.cc:1905] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:43:31.258656 8262 ts_tablet_manager.cc:1918] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:43:31.258656 8264 ts_tablet_manager.cc:1918] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:43:31.258767 8262 log.cc:1196] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/7246153b9f134f81878b945aeb501d25
I20260810 14:43:31.258767 8264 log.cc:1196] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/7246153b9f134f81878b945aeb501d25
I20260810 14:43:31.259089 8262 ts_tablet_manager.cc:1939] T 7246153b9f134f81878b945aeb501d25 P fb5125fb1b9941cc99710ee7bd6282f6: Deleting consensus metadata
I20260810 14:43:31.259089 8264 ts_tablet_manager.cc:1939] T 7246153b9f134f81878b945aeb501d25 P f9dd5ae1fae6450a8ce49b16637ddde7: Deleting consensus metadata
I20260810 14:43:31.260465 7410 catalog_manager.cc:4841] TS f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217): tablet 7246153b9f134f81878b945aeb501d25 (table test-table [id=0a2d6854d5a147028c70843fcd2f9e6b]) successfully deleted
I20260810 14:43:31.260489 7412 catalog_manager.cc:4841] TS fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591): tablet 7246153b9f134f81878b945aeb501d25 (table test-table [id=0a2d6854d5a147028c70843fcd2f9e6b]) successfully deleted
W20260810 14:43:31.262020 7412 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:43907 (error 108)
I20260810 14:43:31.262264 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 7772
I20260810 14:43:31.270346 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:31.269891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43907 (local address 127.0.0.1:44708)
0810 14:43:31.270097 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:31.270222 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:43907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 14:43:31.270673 7412 catalog_manager.cc:4584] TS 9b8b2e084cec486c8212030926e3e01e (127.15.84.1:43907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7246153b9f134f81878b945aeb501d25: Network error: Client connection negotiation failed: client connection to 127.15.84.1:43907: connect: Connection refused (error 111)
I20260810 14:43:31.276947 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 7987
I20260810 14:43:31.288120 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 7382
I20260810 14:43:51.303905 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40369
--webserver_interface=127.15.84.62
--webserver_port=37561
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:40369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:43:51.421402 8267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:51.421702 8267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:51.421764 8267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:51.425911 8267 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 14:43:51.425982 8267 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 14:43:51.426003 8267 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 14:43:51.426021 8267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:43:51.426044 8267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:51.426069 8267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:51.426106 8267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:43:51.426134 8267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:43:51.431747 8267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:40369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=37561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:51.432910 8267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:51.434018 8267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:43:51.448261 8267 server_base.cc:1034] running on GCE node
W20260810 14:43:51.448203 8274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:51.448203 8275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:51.448220 8277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:51.449146 8267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:51.450500 8267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:51.451676 8267 hybrid_clock.cc:648] HybridClock initialized: now 1786373031451643 us; error 52 us; skew 500 ppm
Aug 10 14:43:51 dist-test-slave-rmhg krb5kdc[7354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373031, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:51.458595 8267 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:43:51.459703 8267 webserver.cc:460] Webserver started at http://127.15.84.62:37561/ using document root <none> and password file <none>
I20260810 14:43:51.460336 8267 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:51.460429 8267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:51.464774 8267 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:43:51.467311 8291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:51.468688 8267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:43:51.468833 8267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "9df69aa180b24b8eb01b4d41af759d0f"
format_stamp: "Formatted at 2026-08-10 14:43:29 on dist-test-slave-rmhg"
server_key: "8f5d6b9a3390c09984c0ad9f8957cb50"
server_key_iv: "4ab92e5d49d95c4730a983f3415c1e2a"
server_key_version: "encryptionkey@0"
I20260810 14:43:51.469564 8267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:51.493721 8267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:51.497262 8267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:51.497483 8267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:51.505272 8267 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:40369
I20260810 14:43:51.505548 8371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:40369 every 8 connection(s)
I20260810 14:43:51.506520 8267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:43:51.511231 8374 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:43:51.512182 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 8267
I20260810 14:43:51.512634 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:43907
--local_ip_for_outbound_sockets=127.15.84.1
--tserver_master_addrs=127.15.84.62:40369
--webserver_port=34139
--webserver_interface=127.15.84.1
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:43:51.513206 8374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Bootstrap starting.
I20260810 14:43:51.531939 8374 log.cc:824] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:51.541533 8374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:43:51.541940 8374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Bootstrap complete.
I20260810 14:43:51.546308 8374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } }
I20260810 14:43:51.546784 8374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df69aa180b24b8eb01b4d41af759d0f, State: Initialized, Role: FOLLOWER
I20260810 14:43:51.547307 8374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } }
I20260810 14:43:51.547443 8374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:43:51.547520 8374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:43:51.547618 8374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:43:51.550136 8374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } }
I20260810 14:43:51.550506 8374 leader_election.cc:302] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9df69aa180b24b8eb01b4d41af759d0f; no voters:
I20260810 14:43:51.550872 8374 leader_election.cc:288] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 14:43:51.550961 8381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:43:51.551174 8381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 2 LEADER]: Becoming Leader. State: Replica: 9df69aa180b24b8eb01b4d41af759d0f, State: Running, Role: LEADER
I20260810 14:43:51.551574 8381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } }
I20260810 14:43:51.552280 8374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:43:51.552752 8381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9df69aa180b24b8eb01b4d41af759d0f. Latest consensus state: current_term: 2 leader_uuid: "9df69aa180b24b8eb01b4d41af759d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } } }
I20260810 14:43:51.552867 8381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:51.553045 8384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9df69aa180b24b8eb01b4d41af759d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } } }
I20260810 14:43:51.553186 8384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: This master's current role is: LEADER
I20260810 14:43:51.553475 8389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:43:51.558852 8389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:43:51.559628 8389 catalog_manager.cc:1259] Loaded cluster ID: 44ebccd549674e85bfa510a806d6d444
I20260810 14:43:51.559700 8389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:43:51.562386 8389 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:43:51.563270 8389 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Loaded TSK: 0
I20260810 14:43:51.589871 8389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Generated new TSK 1
I20260810 14:43:51.590216 8389 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:43:51.639056 8378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:51.639410 8378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:51.639478 8378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:51.643728 8378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:51.643810 8378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:51.643895 8378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:43:51.649672 8378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:43907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=34139
--tserver_master_addrs=127.15.84.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:51.650796 8378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:51.651868 8378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:51.662355 8413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:51.662395 8414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:51.662364 8417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:51.662869 8378 server_base.cc:1034] running on GCE node
I20260810 14:43:51.663405 8378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:51.664319 8378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:51.665525 8378 hybrid_clock.cc:648] HybridClock initialized: now 1786373031665499 us; error 40 us; skew 500 ppm
Aug 10 14:43:51 dist-test-slave-rmhg krb5kdc[7354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373031, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:51.669304 8378 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:43:51.670714 8378 webserver.cc:460] Webserver started at http://127.15.84.1:34139/ using document root <none> and password file <none>
I20260810 14:43:51.671545 8378 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:51.671677 8378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:51.677698 8378 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:43:51.680855 8427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:51.682678 8378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:43:51.682852 8378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "9b8b2e084cec486c8212030926e3e01e"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "b0cfd9195d069754b11749d11afcc624"
server_key_iv: "fd4a8c3cd407b068b67c559f8eaf63d5"
server_key_version: "encryptionkey@0"
I20260810 14:43:51.683704 8378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:51.702092 8378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:51.706719 8378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:51.706959 8378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:51.707871 8378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:51.709473 8435 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 14:43:51.714583 8378 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 14:43:51.714716 8378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 14:43:51.714794 8378 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 14:43:51.717389 8378 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 14:43:51.717468 8378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260810 14:43:51.717785 8435 tablet_bootstrap.cc:492] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Bootstrap starting.
I20260810 14:43:51.728439 8378 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:43907
I20260810 14:43:51.728544 8623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:43907 every 8 connection(s)
I20260810 14:43:51.729681 8378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:43:51.731153 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 8378
I20260810 14:43:51.731709 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:38591
--local_ip_for_outbound_sockets=127.15.84.2
--tserver_master_addrs=127.15.84.62:40369
--webserver_port=37865
--webserver_interface=127.15.84.2
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:43:51.739621 8435 log.cc:824] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Log is configured to *not* fsync() on all Append() calls
Aug 10 14:43:51 dist-test-slave-rmhg krb5kdc[7354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373031, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:51.749214 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:51.731837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:56923 (local address 127.15.84.62:40369)
0810 14:43:51.732395 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:51.732417 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:51.733211 (+ 794us) server_negotiation.cc:407] Connection header received
0810 14:43:51.734362 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:51.734380 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:51.734902 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:51.735393 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:51.736598 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:51.737747 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:51.739089 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:51.739597 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:51.739764 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:51.742739 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:51.742747 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:51.742768 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:51.742812 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:51.745113 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:51.745845 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:51.745853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:51.745859 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:51.746075 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:51.746485 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:51.746489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:51.746491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:51.747075 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:51.747239 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:51.747919 (+ 680us) server_negotiation.cc:299] Negotiation successful
0810 14:43:51.748285 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":146,"threads_started":1}
I20260810 14:43:51.749224 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:51.731869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:56923)
0810 14:43:51.732725 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:51.732787 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:51.733799 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:51.735446 (+ 1647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:51.735454 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:51.735850 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:51.736401 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:51.736417 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:51.737915 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:51.737918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:51.738941 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:51.738953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:51.739747 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:51.739752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:51.739971 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:51.740570 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:51.740592 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:51.742452 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:51.745283 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:51.745290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:51.745305 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:51.745690 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:51.746191 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:51.746194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:51.746196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:51.746378 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:51.747215 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:51.747221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:51.747548 (+ 327us) client_negotiation.cc:769] Sending connection context
0810 14:43:51.747752 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 14:43:51.748130 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":186,"threads_started":1}
I20260810 14:43:51.749938 8435 tablet_bootstrap.cc:492] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:43:51.750438 8435 tablet_bootstrap.cc:492] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Bootstrap complete.
I20260810 14:43:51.751673 8435 ts_tablet_manager.cc:1397] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.009s
I20260810 14:43:51.752369 8624 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40369
I20260810 14:43:51.752641 8624 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:51.753347 8624 heartbeater.cc:511] Master 127.15.84.62:40369 requested a full tablet report, sending...
I20260810 14:43:51.754980 8435 raft_consensus.cc:348] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } }
I20260810 14:43:51.755705 8435 raft_consensus.cc:729] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8b2e084cec486c8212030926e3e01e, State: Initialized, Role: FOLLOWER
I20260810 14:43:51.756313 8435 consensus_queue.cc:260] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } }
I20260810 14:43:51.757193 8315 ts_manager.cc:194] Registered new tserver with Master: 9b8b2e084cec486c8212030926e3e01e (127.15.84.1:43907)
I20260810 14:43:51.757560 8435 ts_tablet_manager.cc:1428] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 14:43:51.760838 8315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:56923
I20260810 14:43:51.764190 8624 heartbeater.cc:503] Master 127.15.84.62:40369 was elected leader, sending a full tablet report...
Aug 10 14:43:51 dist-test-slave-rmhg krb5kdc[7354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373031, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for kudu/127.15.84.1@KRBTEST.COM
I20260810 14:43:51.771654 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:51.760545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.15.84.1:43907)
0810 14:43:51.761385 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:51.761398 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:51.761426 (+ 28us) server_negotiation.cc:407] Connection header received
0810 14:43:51.761521 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:51.761526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:51.761751 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:51.761903 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:51.762701 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:51.763550 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:51.764483 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:51.764917 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:51.765024 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:51.767720 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:51.767730 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:51.767742 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:51.767769 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:51.769504 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:51.770137 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:51.770142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:51.770143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:51.770238 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:51.770518 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:51.770520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:51.770521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:51.770814 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:51.770925 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:51.771310 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 14:43:51.771473 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":733,"thread_start_us":120,"threads_started":1}
I20260810 14:43:51.771663 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:51.760468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43907 (local address 127.0.0.1:52314)
0810 14:43:51.760948 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:51.761019 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:51.761205 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:51.761933 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:51.761940 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:51.762430 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:51.762566 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:51.762574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:51.763681 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:51.763685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:51.764354 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:51.764367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:51.765038 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:51.765042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:51.765247 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:51.765801 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:51.765814 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:51.767540 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:51.769637 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:51.769652 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:51.769658 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:51.769998 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:51.770324 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:51.770327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:51.770329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:51.770432 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:51.770924 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:51.770930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:51.771085 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 14:43:51.771235 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 14:43:51.771473 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":129,"threads_started":1}
I20260810 14:43:51.772789 8508 tablet_service.cc:1514] Processing DeleteTablet for tablet 7246153b9f134f81878b945aeb501d25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:31 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:52314
I20260810 14:43:51.773787 8640 tablet_replica.cc:330] stopping tablet replica
I20260810 14:43:51.774248 8640 raft_consensus.cc:2229] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:43:51.774560 8640 raft_consensus.cc:2258] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:43:51.775796 8640 ts_tablet_manager.cc:1905] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:43:51.781077 8640 ts_tablet_manager.cc:1918] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:43:51.781260 8640 log.cc:1196] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal/wals/7246153b9f134f81878b945aeb501d25
I20260810 14:43:51.781566 8640 ts_tablet_manager.cc:1939] T 7246153b9f134f81878b945aeb501d25 P 9b8b2e084cec486c8212030926e3e01e: Deleting consensus metadata
I20260810 14:43:51.784055 8296 catalog_manager.cc:4841] TS 9b8b2e084cec486c8212030926e3e01e (127.15.84.1:43907): tablet 7246153b9f134f81878b945aeb501d25 (table test-table [id=0a2d6854d5a147028c70843fcd2f9e6b]) successfully deleted
W20260810 14:43:51.868044 8630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:51.868327 8630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:51.868371 8630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:51.872578 8630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:51.872658 8630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:51.872720 8630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:43:51.878490 8630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:38591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=37865
--tserver_master_addrs=127.15.84.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:51.879637 8630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:51.880720 8630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:51.890842 8647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:51.890846 8650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:51.891008 8630 server_base.cc:1034] running on GCE node
W20260810 14:43:51.891155 8648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:51.891551 8630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:51.892241 8630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:51.893402 8630 hybrid_clock.cc:648] HybridClock initialized: now 1786373031893369 us; error 46 us; skew 500 ppm
Aug 10 14:43:51 dist-test-slave-rmhg krb5kdc[7354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373031, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:51.896317 8630 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:43:51.897361 8630 webserver.cc:460] Webserver started at http://127.15.84.2:37865/ using document root <none> and password file <none>
I20260810 14:43:51.897928 8630 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:51.898005 8630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:51.902060 8630 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:43:51.904287 8659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:51.905787 8630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:43:51.905910 8630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "fb5125fb1b9941cc99710ee7bd6282f6"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "503a360583b1c291a3e05693f1f74606"
server_key_iv: "623010ce544cd964a6a19502ece517f1"
server_key_version: "encryptionkey@0"
I20260810 14:43:51.906504 8630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:51.936537 8630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:51.939574 8630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:51.939766 8630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:51.940448 8630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:51.941403 8630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:51.941455 8630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:51.941505 8630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:51.941520 8630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:51.950807 8630 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:38591
I20260810 14:43:51.950835 8851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:38591 every 8 connection(s)
I20260810 14:43:51.951995 8630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
Aug 10 14:43:51 dist-test-slave-rmhg krb5kdc[7354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373031, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:51.962301 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 8630
I20260810 14:43:51.962764 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:40217
--local_ip_for_outbound_sockets=127.15.84.3
--tserver_master_addrs=127.15.84.62:40369
--webserver_port=44875
--webserver_interface=127.15.84.3
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:43:51.966773 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:51.954065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:57451 (local address 127.15.84.62:40369)
0810 14:43:51.954240 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:51.954244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:51.955190 (+ 946us) server_negotiation.cc:407] Connection header received
0810 14:43:51.956589 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:51.956593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:51.956648 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:51.956731 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:51.958167 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:51.958709 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:51.959840 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:51.960070 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:51.960154 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:51.962731 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:51.962736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:51.962739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:51.962774 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:51.964248 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:51.964901 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:51.964904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:51.964905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:51.964988 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:51.965404 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:51.965407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:51.965408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:51.965604 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:51.965685 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:51.966451 (+ 766us) server_negotiation.cc:299] Negotiation successful
0810 14:43:51.966601 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 14:43:51.968087 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:51.954532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:57451)
0810 14:43:51.955017 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:51.955069 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:51.956344 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:51.956908 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:51.956915 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:51.957424 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:51.958014 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:51.958030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:51.958816 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:51.958819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:51.959741 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:51.959751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:51.960221 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:51.960225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:51.960382 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:51.960936 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:51.960953 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:51.962538 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:51.964388 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:51.964394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:51.964409 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:51.964794 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:51.965076 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:51.965079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:51.965081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:51.965321 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:51.965702 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:51.965709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:51.966146 (+ 437us) client_negotiation.cc:769] Sending connection context
0810 14:43:51.966378 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 14:43:51.966982 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":133,"threads_started":1}
I20260810 14:43:51.969761 8852 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40369
I20260810 14:43:51.970017 8852 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:51.970566 8852 heartbeater.cc:511] Master 127.15.84.62:40369 requested a full tablet report, sending...
I20260810 14:43:51.971742 8315 ts_manager.cc:194] Registered new tserver with Master: fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591)
I20260810 14:43:51.972349 8315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:57451
W20260810 14:43:52.081326 8859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:43:52.081621 8859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:43:52.081691 8859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:43:52.085790 8859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:43:52.085866 8859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:43:52.085952 8859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:43:52.091799 8859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:40217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=44875
--tserver_master_addrs=127.15.84.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:43:52.092949 8859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:43:52.094069 8859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:43:52.104136 8865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:52.104146 8872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:43:52.104163 8870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:43:52.104583 8859 server_base.cc:1034] running on GCE node
I20260810 14:43:52.105016 8859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:43:52.105759 8859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:43:52.106952 8859 hybrid_clock.cc:648] HybridClock initialized: now 1786373032106918 us; error 47 us; skew 500 ppm
Aug 10 14:43:52 dist-test-slave-rmhg krb5kdc[7354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373032, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:43:52.109881 8859 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:43:52.110889 8859 webserver.cc:460] Webserver started at http://127.15.84.3:44875/ using document root <none> and password file <none>
I20260810 14:43:52.111471 8859 fs_manager.cc:359] Metadata directory not provided
I20260810 14:43:52.111552 8859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:43:52.115587 8859 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 14:43:52.117779 8883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:52.119092 8859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:43:52.119232 8859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "f9dd5ae1fae6450a8ce49b16637ddde7"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "641117d730c0211ba3f481b2fca4e27a"
server_key_iv: "96694a86fd0e30235ae972fa76e5a47c"
server_key_version: "encryptionkey@0"
I20260810 14:43:52.119777 8859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:43:52.146636 8859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:43:52.150090 8859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:43:52.150306 8859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:43:52.150960 8859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:43:52.152034 8859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:43:52.152096 8859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:52.152166 8859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:43:52.152194 8859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:43:52.161253 8859 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:40217
I20260810 14:43:52.161289 9056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:40217 every 8 connection(s)
I20260810 14:43:52.162318 8859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
Aug 10 14:43:52 dist-test-slave-rmhg krb5kdc[7354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373032, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:43:52.172336 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 8859
I20260810 14:43:52.176934 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.164241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:35857 (local address 127.15.84.62:40369)
0810 14:43:52.164376 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:52.164380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:52.165380 (+ 1000us) server_negotiation.cc:407] Connection header received
0810 14:43:52.166796 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:52.166799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:52.166861 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:52.166937 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:52.168228 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.168778 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.170003 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.170243 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.170340 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:52.173052 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:52.173056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:52.173059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:52.173095 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:52.174711 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:52.175225 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:52.175229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:52.175231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:52.175319 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:52.175749 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:52.175751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:52.175752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:52.175926 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:52.176001 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:52.176632 (+ 631us) server_negotiation.cc:299] Negotiation successful
0810 14:43:52.176767 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:43:52.178332 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.164678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:35857)
0810 14:43:52.165227 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:52.165276 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:52.166546 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:52.167145 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:52.167154 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:52.167527 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:43:52.168048 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:52.168063 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.168911 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.168914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:52.169875 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:52.169887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.170402 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.170406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:52.170581 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:52.171184 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:43:52.171201 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:43:52.172856 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:43:52.174834 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:52.174840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:52.174855 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:52.175139 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:52.175414 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:43:52.175417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:43:52.175419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:43:52.175657 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:43:52.176019 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:43:52.176026 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:43:52.176378 (+ 352us) client_negotiation.cc:769] Sending connection context
0810 14:43:52.176576 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 14:43:52.176915 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":142,"threads_started":1}
I20260810 14:43:52.179772 9057 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40369
I20260810 14:43:52.180099 9057 heartbeater.cc:464] Registering TS with master...
I20260810 14:43:52.180750 9057 heartbeater.cc:511] Master 127.15.84.62:40369 requested a full tablet report, sending...
I20260810 14:43:52.181746 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.173074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42478 (local address 127.15.84.62:40369)
0810 14:43:52.173345 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:52.173349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:52.173532 (+ 183us) server_negotiation.cc:407] Connection header received
0810 14:43:52.173671 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:52.173674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:52.173728 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:52.173822 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:52.174714 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.175320 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.176425 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.176705 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.176805 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:52.178328 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:52.178331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:52.178333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:52.178355 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:52.179812 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:52.180290 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:52.180293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:52.180294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:52.180371 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:52.180633 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:52.180636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:52.180638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:52.180797 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:52.180976 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:52.181210 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 14:43:52.181332 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":106,"threads_started":1}
I20260810 14:43:52.182061 8315 ts_manager.cc:194] Registered new tserver with Master: f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217)
I20260810 14:43:52.182714 8315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:35857
I20260810 14:43:52.183525 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:43:52.187642 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.184152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42492 (local address 127.15.84.62:40369)
0810 14:43:52.184260 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:52.184264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:52.184482 (+ 218us) server_negotiation.cc:407] Connection header received
0810 14:43:52.184632 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:52.184642 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:52.184740 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:52.184824 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:43:52.185063 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.185618 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.186488 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.186728 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.186817 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:52.187121 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:43:52.187256 (+ 135us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 14:43:52.187503 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42492: token expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 14:43:52.187718 9064 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42492: token expired
W20260810 14:43:52.188041 8206 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.15.84.62:40369)
I20260810 14:43:52.195834 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.188399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.15.84.62:40369)
0810 14:43:52.188500 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:52.188504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:52.188517 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:43:52.188606 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:52.188608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:52.188648 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:52.188708 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:43:52.189287 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.189817 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.190628 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.190869 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.190945 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:52.192177 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:43:52.192180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:43:52.192182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:43:52.192205 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:43:52.193919 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:52.194431 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:52.194435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:52.194437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:52.194539 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:43:52.194804 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:43:52.194807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:43:52.194809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:43:52.195049 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:43:52.195193 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:52.195500 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 14:43:52.195668 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 14:43:52.198505 8313 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:43:52.201861 8313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:43:52.210376 8508 tablet_service.cc:1467] Processing CreateTablet for tablet 62c607cc69ae421793ad6365fec185f3 (DEFAULT_TABLE table=test-table [id=eab77f9fd3cb45e4a27bc20eec2d4e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:52.211112 8508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c607cc69ae421793ad6365fec185f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:52.214028 8435 tablet_bootstrap.cc:492] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e: Bootstrap starting.
I20260810 14:43:52.214802 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.209330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:38591 (local address 127.0.0.1:34364)
0810 14:43:52.209609 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:52.209623 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:52.209748 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:52.210507 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:52.210510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:52.210529 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:52.210680 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:52.210687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.212032 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.212035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:52.213136 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:52.213186 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.214327 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.214331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:52.214438 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:52.214502 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 14:43:52.214557 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 14:43:52.214652 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":3,"thread_start_us":130,"threads_started":1}
I20260810 14:43:52.214960 8435 tablet_bootstrap.cc:654] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:52.215368 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.209094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:40217 (local address 127.0.0.1:43356)
0810 14:43:52.209250 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:52.209267 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:52.209363 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:52.210539 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:52.210542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:52.210563 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:52.211135 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:52.211145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.212550 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.212553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:52.213550 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:52.213559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.215005 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.215009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:52.215123 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:52.215151 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 14:43:52.215197 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 14:43:52.215266 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":407}
I20260810 14:43:52.215626 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.209550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.15.84.2:38591)
0810 14:43:52.210057 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:52.210075 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:52.210096 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:43:52.210168 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:52.210173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:52.210352 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:52.210503 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:52.210793 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.211924 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.213298 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.214186 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.215104 (+ 918us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:52.215289 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:52.215356 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 14:43:52.215462 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":115,"threads_started":1}
I20260810 14:43:52.216126 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.209612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.15.84.3:40217)
0810 14:43:52.210057 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:52.210071 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:52.210096 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:43:52.210174 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:52.210179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:52.210376 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:52.210657 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:52.211302 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.212421 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.213677 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.214860 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.215651 (+ 791us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:52.215818 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:52.215874 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 14:43:52.215979 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":143,"threads_started":1}
I20260810 14:43:52.216739 8435 tablet_bootstrap.cc:492] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e: No bootstrap required, opened a new log
I20260810 14:43:52.216835 8435 ts_tablet_manager.cc:1397] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:43:52.217344 8435 raft_consensus.cc:348] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:43:52.217468 8435 raft_consensus.cc:374] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:52.217501 8435 raft_consensus.cc:729] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8b2e084cec486c8212030926e3e01e, State: Initialized, Role: FOLLOWER
I20260810 14:43:52.217629 8435 consensus_queue.cc:260] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:43:52.218185 8435 ts_tablet_manager.cc:1428] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 14:43:52.219532 8958 tablet_service.cc:1467] Processing CreateTablet for tablet 62c607cc69ae421793ad6365fec185f3 (DEFAULT_TABLE table=test-table [id=eab77f9fd3cb45e4a27bc20eec2d4e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:52.219528 8741 tablet_service.cc:1467] Processing CreateTablet for tablet 62c607cc69ae421793ad6365fec185f3 (DEFAULT_TABLE table=test-table [id=eab77f9fd3cb45e4a27bc20eec2d4e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:43:52.220717 8741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c607cc69ae421793ad6365fec185f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:52.220723 8958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c607cc69ae421793ad6365fec185f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:43:52.230050 9083 tablet_bootstrap.cc:492] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7: Bootstrap starting.
I20260810 14:43:52.229871 9082 tablet_bootstrap.cc:492] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6: Bootstrap starting.
W20260810 14:43:52.230947 8625 tablet.cc:2369] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:43:52.232486 9083 tablet_bootstrap.cc:654] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:52.232975 9082 tablet_bootstrap.cc:654] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6: Neither blocks nor log segments found. Creating new log.
I20260810 14:43:52.233882 9083 log.cc:824] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:52.233882 9082 log.cc:824] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6: Log is configured to *not* fsync() on all Append() calls
I20260810 14:43:52.236171 9083 tablet_bootstrap.cc:492] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7: No bootstrap required, opened a new log
I20260810 14:43:52.236171 9082 tablet_bootstrap.cc:492] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6: No bootstrap required, opened a new log
I20260810 14:43:52.236393 9083 ts_tablet_manager.cc:1397] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 14:43:52.236394 9082 ts_tablet_manager.cc:1397] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 14:43:52.240495 9083 raft_consensus.cc:348] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:43:52.240495 9082 raft_consensus.cc:348] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:43:52.240823 9083 raft_consensus.cc:374] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:52.240823 9082 raft_consensus.cc:374] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:43:52.240883 9082 raft_consensus.cc:729] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5125fb1b9941cc99710ee7bd6282f6, State: Initialized, Role: FOLLOWER
I20260810 14:43:52.240891 9083 raft_consensus.cc:729] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9dd5ae1fae6450a8ce49b16637ddde7, State: Initialized, Role: FOLLOWER
I20260810 14:43:52.241425 9082 consensus_queue.cc:260] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:43:52.241425 9083 consensus_queue.cc:260] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:43:52.242159 8852 heartbeater.cc:503] Master 127.15.84.62:40369 was elected leader, sending a full tablet report...
I20260810 14:43:52.242547 9082 ts_tablet_manager.cc:1428] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 14:43:52.242547 9083 ts_tablet_manager.cc:1428] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 14:43:52.242776 9057 heartbeater.cc:503] Master 127.15.84.62:40369 was elected leader, sending a full tablet report...
I20260810 14:43:52.315744 9089 raft_consensus.cc:482] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:43:52.315955 9089 raft_consensus.cc:504] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:43:52.317068 9089 leader_election.cc:288] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591), 9b8b2e084cec486c8212030926e3e01e (127.15.84.1:43907)
I20260810 14:43:52.321822 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.317363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:38591 (local address 127.15.84.3:35409)
0810 14:43:52.317499 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:52.317515 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:52.317615 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:52.317951 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:52.317954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:52.317975 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:52.318105 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:52.318112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.319116 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.319118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:52.320398 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:52.320410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.321351 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.321354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:52.321469 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:52.321497 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 14:43:52.321575 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:43:52.321671 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":5}
I20260810 14:43:52.321843 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.317451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:35409 (local address 127.15.84.2:38591)
0810 14:43:52.317601 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:52.317606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:52.317625 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:43:52.317726 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:52.317729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:52.317777 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:52.317857 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:52.318233 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.319003 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.320528 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.321222 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.321333 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:52.321410 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:52.321622 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 14:43:52.321711 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 14:43:52.321846 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.317795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:50005 (local address 127.15.84.1:43907)
0810 14:43:52.318071 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:52.318074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:52.318087 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:43:52.318166 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:52.318169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:52.318235 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:52.318361 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:43:52.318574 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.319262 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.320621 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.321326 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.321419 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:52.321613 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:52.321657 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 14:43:52.321722 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":117,"threads_started":1}
I20260810 14:43:52.321830 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.317638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43907 (local address 127.15.84.3:50005)
0810 14:43:52.317995 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:52.318009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:43:52.318088 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:43:52.318370 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:43:52.318372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:43:52.318387 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:43:52.318479 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:52.318483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.319367 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.319370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:52.320505 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:43:52.320513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.321438 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.321441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:43:52.321529 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:52.321548 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 14:43:52.321598 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 14:43:52.321702 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":39,"thread_start_us":123,"threads_started":1}
I20260810 14:43:52.322585 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c607cc69ae421793ad6365fec185f3" candidate_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" is_pre_election: true
I20260810 14:43:52.322782 8542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c607cc69ae421793ad6365fec185f3" candidate_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b8b2e084cec486c8212030926e3e01e" is_pre_election: true
I20260810 14:43:52.322986 8773 raft_consensus.cc:2454] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9dd5ae1fae6450a8ce49b16637ddde7 in term 0.
I20260810 14:43:52.323065 8542 raft_consensus.cc:2454] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9dd5ae1fae6450a8ce49b16637ddde7 in term 0.
I20260810 14:43:52.323487 8888 leader_election.cc:302] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f9dd5ae1fae6450a8ce49b16637ddde7, fb5125fb1b9941cc99710ee7bd6282f6; no voters:
I20260810 14:43:52.323760 9089 raft_consensus.cc:2781] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:43:52.323861 9089 raft_consensus.cc:482] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:43:52.323894 9089 raft_consensus.cc:3037] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:52.325014 9089 raft_consensus.cc:504] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:43:52.325367 9089 leader_election.cc:288] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [CANDIDATE]: Term 1 election: Requested vote from peers fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591), 9b8b2e084cec486c8212030926e3e01e (127.15.84.1:43907)
I20260810 14:43:52.325824 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c607cc69ae421793ad6365fec185f3" candidate_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5125fb1b9941cc99710ee7bd6282f6"
I20260810 14:43:52.325819 8542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c607cc69ae421793ad6365fec185f3" candidate_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b8b2e084cec486c8212030926e3e01e"
I20260810 14:43:52.325940 8773 raft_consensus.cc:3037] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:52.325945 8542 raft_consensus.cc:3037] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:43:52.326954 8542 raft_consensus.cc:2454] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9dd5ae1fae6450a8ce49b16637ddde7 in term 1.
I20260810 14:43:52.326975 8773 raft_consensus.cc:2454] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9dd5ae1fae6450a8ce49b16637ddde7 in term 1.
I20260810 14:43:52.327226 8888 leader_election.cc:302] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b8b2e084cec486c8212030926e3e01e, f9dd5ae1fae6450a8ce49b16637ddde7; no voters:
I20260810 14:43:52.327415 9089 raft_consensus.cc:2781] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:43:52.327646 9089 raft_consensus.cc:686] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 1 LEADER]: Becoming Leader. State: Replica: f9dd5ae1fae6450a8ce49b16637ddde7, State: Running, Role: LEADER
I20260810 14:43:52.327975 9089 consensus_queue.cc:237] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:43:52.331243 8315 catalog_manager.cc:5495] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3). New cstate: current_term: 1 leader_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } health_report { overall_health: UNKNOWN } } }
I20260810 14:43:52.346827 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.343643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.15.84.3:40217)
0810 14:43:52.343762 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 14:43:52.343765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:43:52.343793 (+ 28us) server_negotiation.cc:407] Connection header received
0810 14:43:52.343870 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:43:52.343873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:43:52.343929 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:43:52.344012 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:43:52.344193 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.344715 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.345537 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:43:52.345750 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:43:52.345826 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:43:52.346057 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:43:52.346499 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:43:52.346589 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:43:52.346622 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 14:43:52.346676 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:43:52.353070 8542 raft_consensus.cc:1261] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Refusing update from remote peer f9dd5ae1fae6450a8ce49b16637ddde7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:43:52.353080 8773 raft_consensus.cc:1261] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6 [term 1 FOLLOWER]: Refusing update from remote peer f9dd5ae1fae6450a8ce49b16637ddde7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:43:52.353871 9089 consensus_queue.cc:1035] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:43:52.354090 9093 consensus_queue.cc:1035] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:43:52.359323 9096 mvcc.cc:205] Tried to move back new op lower bound from 7316983940510531584 to 7316983940417990656. Current Snapshot: MvccSnapshot[applied={T|T < 7316983940510531584}]
I20260810 14:43:52.365528 9098 mvcc.cc:205] Tried to move back new op lower bound from 7316983940510531584 to 7316983940417990656. Current Snapshot: MvccSnapshot[applied={T|T < 7316983940510531584 or (T in {7316983940510531584})}]
I20260810 14:43:52.368577 8315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:43:52.368793 8315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:43:52.371649 8315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Sending DeleteTablet for 3 replicas of tablet 62c607cc69ae421793ad6365fec185f3
I20260810 14:43:52.372466 8508 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c607cc69ae421793ad6365fec185f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:52 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:52314
I20260810 14:43:52.372501 8958 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c607cc69ae421793ad6365fec185f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:52 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:43356
I20260810 14:43:52.372558 8741 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c607cc69ae421793ad6365fec185f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:43:52 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:34364
I20260810 14:43:52.373198 9109 tablet_replica.cc:330] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e: stopping tablet replica
I20260810 14:43:52.373221 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 8378
I20260810 14:43:52.398543 9110 tablet_replica.cc:330] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6: stopping tablet replica
I20260810 14:43:52.399031 9110 raft_consensus.cc:2229] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:43:52.399490 9110 raft_consensus.cc:2258] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:43:52.399971 9112 tablet_replica.cc:330] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7: stopping tablet replica
I20260810 14:43:52.400321 9112 raft_consensus.cc:2229] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:43:52.400811 9112 raft_consensus.cc:2258] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:43:52.402724 9112 ts_tablet_manager.cc:1905] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:43:52.402724 9110 ts_tablet_manager.cc:1905] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:43:52.405778 9110 ts_tablet_manager.cc:1918] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:43:52.405786 9112 ts_tablet_manager.cc:1918] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:43:52.405920 9110 log.cc:1196] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/62c607cc69ae421793ad6365fec185f3
I20260810 14:43:52.406069 9112 log.cc:1196] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/62c607cc69ae421793ad6365fec185f3
I20260810 14:43:52.406306 9110 ts_tablet_manager.cc:1939] T 62c607cc69ae421793ad6365fec185f3 P fb5125fb1b9941cc99710ee7bd6282f6: Deleting consensus metadata
I20260810 14:43:52.406569 9112 ts_tablet_manager.cc:1939] T 62c607cc69ae421793ad6365fec185f3 P f9dd5ae1fae6450a8ce49b16637ddde7: Deleting consensus metadata
W20260810 14:43:52.407083 8296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:43907 (error 108)
I20260810 14:43:52.407819 8296 catalog_manager.cc:4841] TS fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591): tablet 62c607cc69ae421793ad6365fec185f3 (table test-table [id=eab77f9fd3cb45e4a27bc20eec2d4e87]) successfully deleted
I20260810 14:43:52.407878 8292 catalog_manager.cc:4841] TS f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217): tablet 62c607cc69ae421793ad6365fec185f3 (table test-table [id=eab77f9fd3cb45e4a27bc20eec2d4e87]) successfully deleted
I20260810 14:43:52.408437 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 8630
I20260810 14:43:52.411423 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:43:52.408508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:43907 (local address 127.0.0.1:52320)
0810 14:43:52.411232 (+ 2724us) negotiation.cc:107] Waiting for socket to connect
0810 14:43:52.411324 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:43907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2619}
W20260810 14:43:52.411609 8296 catalog_manager.cc:4584] TS 9b8b2e084cec486c8212030926e3e01e (127.15.84.1:43907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62c607cc69ae421793ad6365fec185f3: Network error: Client connection negotiation failed: client connection to 127.15.84.1:43907: connect: Connection refused (error 111)
I20260810 14:43:52.422788 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 8859
I20260810 14:43:52.439177 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 8267
I20260810 14:44:12.461887 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40369
--webserver_interface=127.15.84.62
--webserver_port=37561
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:40369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:12.577374 9114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:12.577682 9114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:12.577723 9114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:12.582017 9114 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 14:44:12.582095 9114 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 14:44:12.582109 9114 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 14:44:12.582122 9114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:12.582140 9114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:12.582175 9114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:12.582195 9114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:12.582226 9114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:12.588045 9114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:40369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=37561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:12.589332 9114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:12.590404 9114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:12.599395 9128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:12.599473 9127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:12.599520 9130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:12.599452 9114 server_base.cc:1034] running on GCE node
I20260810 14:44:12.600150 9114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:12.601428 9114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:12.602608 9114 hybrid_clock.cc:648] HybridClock initialized: now 1786373052602585 us; error 38 us; skew 500 ppm
Aug 10 14:44:12 dist-test-slave-rmhg krb5kdc[7354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373052, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:12.605662 9114 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:44:12.606676 9114 webserver.cc:460] Webserver started at http://127.15.84.62:37561/ using document root <none> and password file <none>
I20260810 14:44:12.607283 9114 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:12.607357 9114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:12.611467 9114 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:44:12.613664 9140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:12.614966 9114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:44:12.615090 9114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "9df69aa180b24b8eb01b4d41af759d0f"
format_stamp: "Formatted at 2026-08-10 14:43:29 on dist-test-slave-rmhg"
server_key: "8f5d6b9a3390c09984c0ad9f8957cb50"
server_key_iv: "4ab92e5d49d95c4730a983f3415c1e2a"
server_key_version: "encryptionkey@0"
I20260810 14:44:12.615657 9114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:12.643626 9114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:12.647027 9114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:12.647269 9114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:12.654779 9114 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:40369
I20260810 14:44:12.655073 9231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:40369 every 8 connection(s)
I20260810 14:44:12.655938 9114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:12.660372 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 9114
I20260810 14:44:12.660403 9232 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:44:12.660789 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:43907
--local_ip_for_outbound_sockets=127.15.84.1
--tserver_master_addrs=127.15.84.62:40369
--webserver_port=34139
--webserver_interface=127.15.84.1
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:44:12.662320 9232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Bootstrap starting.
I20260810 14:44:12.685657 9232 log.cc:824] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:12.697466 9232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:44:12.697867 9232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Bootstrap complete.
I20260810 14:44:12.702085 9232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } }
I20260810 14:44:12.702533 9232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df69aa180b24b8eb01b4d41af759d0f, State: Initialized, Role: FOLLOWER
I20260810 14:44:12.703100 9232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } }
I20260810 14:44:12.703248 9232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:12.703326 9232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:12.703433 9232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 2 FOLLOWER]: Advancing to term 3
I20260810 14:44:12.705673 9232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } }
I20260810 14:44:12.706039 9232 leader_election.cc:302] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9df69aa180b24b8eb01b4d41af759d0f; no voters:
I20260810 14:44:12.706388 9232 leader_election.cc:288] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 14:44:12.706508 9238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 3 FOLLOWER]: Leader election won for term 3
I20260810 14:44:12.706722 9238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [term 3 LEADER]: Becoming Leader. State: Replica: 9df69aa180b24b8eb01b4d41af759d0f, State: Running, Role: LEADER
I20260810 14:44:12.707175 9238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } }
I20260810 14:44:12.707832 9232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:12.708650 9240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9df69aa180b24b8eb01b4d41af759d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } } }
I20260810 14:44:12.708801 9240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:12.709102 9241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9df69aa180b24b8eb01b4d41af759d0f. Latest consensus state: current_term: 3 leader_uuid: "9df69aa180b24b8eb01b4d41af759d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df69aa180b24b8eb01b4d41af759d0f" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40369 } } }
I20260810 14:44:12.709241 9241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:12.709765 9247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:12.715559 9247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:12.716249 9247 catalog_manager.cc:1259] Loaded cluster ID: 44ebccd549674e85bfa510a806d6d444
I20260810 14:44:12.716318 9247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:12.718945 9247 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:12.719759 9247 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Loaded TSK: 0
I20260810 14:44:12.719847 9247 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9df69aa180b24b8eb01b4d41af759d0f: Loaded TSK: 1
F20260810 14:44:12.720605 9247 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7faef506d9ad google::LogMessage::Fail() at ??:0
@ 0x7faef507088a google::LogMessage::SendToLog() at ??:0
@ 0x7faef506d4de google::LogMessage::Flush() at ??:0
@ 0x7faef506ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7faef7d3118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7faef7d29f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7faeff9d27bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7faeff9ac99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7faeff9ad178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7faeff9eedf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7faeffa0d82b std::function<>::operator()() at ??:0
@ 0x7faeff9acb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7faeff9adf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7faeff9aa579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7faeff9ee640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7faef72bf3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7faef72bfc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7faef72c13f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7faef72b0512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7faef88f46db start_thread at ??:0
@ 0x7faef3b6b61f clone at ??:0
W20260810 14:44:12.809063 9235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:12.809373 9235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:12.809422 9235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:12.813855 9235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:12.813911 9235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:12.813973 9235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:44:12.819923 9235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:43907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=34139
--tserver_master_addrs=127.15.84.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:12.821125 9235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:12.822175 9235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:12.831835 9305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:12.831811 9301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:12.831845 9300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:12.832103 9235 server_base.cc:1034] running on GCE node
I20260810 14:44:12.832742 9235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:12.833446 9235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:12.834623 9235 hybrid_clock.cc:648] HybridClock initialized: now 1786373052834596 us; error 50 us; skew 500 ppm
Aug 10 14:44:12 dist-test-slave-rmhg krb5kdc[7354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373052, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:12.837549 9235 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:44:12.838534 9235 webserver.cc:460] Webserver started at http://127.15.84.1:34139/ using document root <none> and password file <none>
I20260810 14:44:12.839156 9235 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:12.839233 9235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:12.843580 9235 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:44:12.846100 9314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:12.847528 9235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:44:12.847651 9235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "9b8b2e084cec486c8212030926e3e01e"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "b0cfd9195d069754b11749d11afcc624"
server_key_iv: "fd4a8c3cd407b068b67c559f8eaf63d5"
server_key_version: "encryptionkey@0"
I20260810 14:44:12.848217 9235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:12.876763 9235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:12.880390 9235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:12.880590 9235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:12.881302 9235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:12.882571 9330 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 14:44:12.888473 9235 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 14:44:12.888569 9235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260810 14:44:12.888645 9235 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 14:44:12.890651 9235 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 14:44:12.890720 9235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260810 14:44:12.891074 9330 tablet_bootstrap.cc:492] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e: Bootstrap starting.
I20260810 14:44:12.901331 9235 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:43907
I20260810 14:44:12.901598 9501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:43907 every 8 connection(s)
I20260810 14:44:12.902432 9235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:44:12.910228 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 9235
I20260810 14:44:12.910672 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:38591
--local_ip_for_outbound_sockets=127.15.84.2
--tserver_master_addrs=127.15.84.62:40369
--webserver_port=37865
--webserver_interface=127.15.84.2
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:44:12.914654 9330 log.cc:824] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:12.924610 9330 tablet_bootstrap.cc:492] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:44:12.925093 9330 tablet_bootstrap.cc:492] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e: Bootstrap complete.
I20260810 14:44:12.926754 9330 ts_tablet_manager.cc:1397] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e: Time spent bootstrapping tablet: real 0.036s user 0.029s sys 0.004s
I20260810 14:44:12.930228 9330 raft_consensus.cc:348] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:44:12.930827 9330 raft_consensus.cc:729] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8b2e084cec486c8212030926e3e01e, State: Initialized, Role: FOLLOWER
I20260810 14:44:12.931398 9330 consensus_queue.cc:260] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:44:12.932543 9330 ts_tablet_manager.cc:1428] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260810 14:44:12.969269 9504 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:44:12.904506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:43969)
0810 14:44:12.905056 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:12.905113 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:12.905987 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:12.968207 (+ 62220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":147,"threads_started":1}
W20260810 14:44:12.969939 9502 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:40369 (0 consecutive failures): Network error: Failed to ping master at 127.15.84.62:40369: Client connection negotiation failed: client connection to 127.15.84.62:40369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 14:44:12.970772 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:12.970460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:58179)
0810 14:44:12.970604 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:12.970697 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 14:44:12.971516 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:12.971285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:40627)
0810 14:44:12.971397 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:12.971451 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 14:44:12.971697 9502 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:44:13.031055 9505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:13.031327 9505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:13.031373 9505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:13.035341 9505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:13.035398 9505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:13.035456 9505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:44:13.040758 9505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:38591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=37865
--tserver_master_addrs=127.15.84.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:13.041893 9505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:13.042922 9505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:13.052776 9515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:13.052769 9516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:13.052776 9520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:13.052814 9505 server_base.cc:1034] running on GCE node
I20260810 14:44:13.053462 9505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:13.054101 9505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:13.055263 9505 hybrid_clock.cc:648] HybridClock initialized: now 1786373053055244 us; error 32 us; skew 500 ppm
Aug 10 14:44:13 dist-test-slave-rmhg krb5kdc[7354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373053, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:13.058046 9505 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:44:13.058935 9505 webserver.cc:460] Webserver started at http://127.15.84.2:37865/ using document root <none> and password file <none>
I20260810 14:44:13.059490 9505 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:13.059556 9505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:13.063573 9505 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:44:13.065799 9533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:13.067128 9505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:44:13.067291 9505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "fb5125fb1b9941cc99710ee7bd6282f6"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "503a360583b1c291a3e05693f1f74606"
server_key_iv: "623010ce544cd964a6a19502ece517f1"
server_key_version: "encryptionkey@0"
I20260810 14:44:13.067878 9505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:13.099787 9505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:13.103221 9505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:13.103410 9505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:13.104058 9505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:13.104964 9505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:13.105011 9505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:13.105052 9505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:13.105068 9505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:13.113930 9505 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:38591
I20260810 14:44:13.113970 9719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:38591 every 8 connection(s)
I20260810 14:44:13.114961 9505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:44:13.118893 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:13.117064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:54407)
0810 14:44:13.117552 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:13.117669 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":112,"threads_started":1}
I20260810 14:44:13.119109 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 9505
W20260810 14:44:13.119513 9720 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:40369 (0 consecutive failures): Network error: Failed to ping master at 127.15.84.62:40369: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
I20260810 14:44:13.119650 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:40217
--local_ip_for_outbound_sockets=127.15.84.3
--tserver_master_addrs=127.15.84.62:40369
--webserver_port=44875
--webserver_interface=127.15.84.3
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:44:13.120216 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:13.119956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:43199)
0810 14:44:13.120082 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:13.120148 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 14:44:13.121655 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:13.120596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:43981)
0810 14:44:13.121530 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:13.121579 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":824}
W20260810 14:44:13.121835 9720 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:44:13.235937 9724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:13.236222 9724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:13.236281 9724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:13.240293 9724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:13.240352 9724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:13.240411 9724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:44:13.245680 9724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:40217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=44875
--tserver_master_addrs=127.15.84.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:13.246752 9724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:13.247757 9724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:13.257594 9739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:13.257606 9736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:13.257606 9733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:13.257764 9724 server_base.cc:1034] running on GCE node
I20260810 14:44:13.258268 9724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:13.258931 9724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:13.260085 9724 hybrid_clock.cc:648] HybridClock initialized: now 1786373053260060 us; error 34 us; skew 500 ppm
Aug 10 14:44:13 dist-test-slave-rmhg krb5kdc[7354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373053, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:13.277952 9724 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:44:13.279013 9724 webserver.cc:460] Webserver started at http://127.15.84.3:44875/ using document root <none> and password file <none>
I20260810 14:44:13.279616 9724 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:13.279682 9724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:13.283891 9724 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 14:44:13.286154 9750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:13.287577 9724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:44:13.287698 9724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "f9dd5ae1fae6450a8ce49b16637ddde7"
format_stamp: "Formatted at 2026-08-10 14:43:30 on dist-test-slave-rmhg"
server_key: "641117d730c0211ba3f481b2fca4e27a"
server_key_iv: "96694a86fd0e30235ae972fa76e5a47c"
server_key_version: "encryptionkey@0"
I20260810 14:44:13.288293 9724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:13.302116 9724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:13.305328 9724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:13.305518 9724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:13.306175 9724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:13.307060 9724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:13.307109 9724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:13.307165 9724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:13.307196 9724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:13.316550 9724 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:40217
I20260810 14:44:13.316592 9919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:40217 every 8 connection(s)
I20260810 14:44:13.317734 9724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:44:13.321909 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:13.319994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:47437)
0810 14:44:13.320488 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:13.320609 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
W20260810 14:44:13.322441 9920 heartbeater.cc:644] Failed to heartbeat to 127.15.84.62:40369 (0 consecutive failures): Network error: Failed to ping master at 127.15.84.62:40369: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
I20260810 14:44:13.323088 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:13.322863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:34139)
0810 14:44:13.322975 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:13.323029 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 14:44:13.323724 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:13.323550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:60685)
0810 14:44:13.323634 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:13.323675 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 14:44:13.323886 9920 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 14:44:13.327973 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 9724
I20260810 14:44:13.972841 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:13.972344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:58529)
0810 14:44:13.972644 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:13.972718 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
I20260810 14:44:14.122766 9930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:14.122334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:57835)
0810 14:44:14.122578 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:14.122636 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260810 14:44:14.219969 9931 raft_consensus.cc:482] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:44:14.220232 9931 raft_consensus.cc:504] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:44:14.221517 9931 leader_election.cc:288] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217), fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591)
Aug 10 14:44:14 dist-test-slave-rmhg krb5kdc[7354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373052, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.2@KRBTEST.COM
Aug 10 14:44:14 dist-test-slave-rmhg krb5kdc[7354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373052, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.3@KRBTEST.COM
I20260810 14:44:14.237960 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:14.221716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:38591 (local address 127.15.84.1:56233)
0810 14:44:14.222002 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:14.222013 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:14.222121 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:14.224284 (+ 2163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:14.224291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:14.224970 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:14.225843 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:14.225868 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:14.227930 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:14.227934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:14.229644 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:14.229660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:14.230734 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:14.230739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:14.230859 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:14.231554 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:44:14.231567 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:14.233110 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:14.235313 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:14.235320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:14.235334 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:14.235653 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:14.235999 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:14.236002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:14.236004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:14.236161 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:14.236829 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:14.236836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:14.237146 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 14:44:14.237353 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 14:44:14.237759 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":387,"spinlock_wait_cycles":32384,"thread_start_us":83,"threads_started":1}
I20260810 14:44:14.237977 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:14.221933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:56233 (local address 127.15.84.2:38591)
0810 14:44:14.222342 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:14.222364 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:14.222407 (+ 43us) server_negotiation.cc:407] Connection header received
0810 14:44:14.223294 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:14.223300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:14.223779 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:14.224152 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:14.226076 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:14.227722 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:14.229841 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:14.230586 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:14.230778 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:14.233388 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:14.233394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:14.233408 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:14.233441 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:14.235192 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:14.235775 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:14.235780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:14.235785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:14.235907 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:14.236263 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:14.236266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:14.236267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:14.236708 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:14.236837 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:14.237510 (+ 673us) server_negotiation.cc:299] Negotiation successful
0810 14:44:14.237815 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":69,"threads_started":1}
I20260810 14:44:14.237980 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:14.221924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:50163 (local address 127.15.84.3:40217)
0810 14:44:14.222342 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:14.222359 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:14.222407 (+ 48us) server_negotiation.cc:407] Connection header received
0810 14:44:14.223268 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:14.223279 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:14.223779 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:14.224152 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:14.226076 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:14.227722 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:14.229822 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:14.230527 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:14.230677 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:14.233643 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:14.233649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:14.233662 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:14.233693 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:14.235254 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:14.235805 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:14.235810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:14.235814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:14.235938 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:14.236269 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:14.236272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:14.236273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:14.236708 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:14.236837 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:14.237509 (+ 672us) server_negotiation.cc:299] Negotiation successful
0810 14:44:14.237784 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":70,"threads_started":1}
I20260810 14:44:14.237991 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:14.221716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:40217 (local address 127.15.84.1:50163)
0810 14:44:14.221986 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:14.222001 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:14.222158 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:14.224284 (+ 2126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:14.224291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:14.224970 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:14.225843 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:14.225867 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:14.227930 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:14.227934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:14.229664 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:14.229671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:14.230658 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:14.230662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:14.230854 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:14.231513 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:44:14.231531 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:14.233447 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:14.235379 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:14.235383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:14.235385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:14.235683 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:14.236034 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:14.236037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:14.236039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:14.236169 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:14.236829 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:14.236835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:14.237136 (+ 301us) client_negotiation.cc:769] Sending connection context
0810 14:44:14.237355 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 14:44:14.237841 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":433,"spinlock_wait_cycles":79360}
W20260810 14:44:14.240556 9322 leader_election.cc:341] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591): Not found: Tablet not found: 62c607cc69ae421793ad6365fec185f3
W20260810 14:44:14.240713 9315 leader_election.cc:341] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217): Not found: Tablet not found: 62c607cc69ae421793ad6365fec185f3
I20260810 14:44:14.240849 9315 leader_election.cc:302] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b8b2e084cec486c8212030926e3e01e; no voters: f9dd5ae1fae6450a8ce49b16637ddde7, fb5125fb1b9941cc99710ee7bd6282f6
I20260810 14:44:14.241247 9931 raft_consensus.cc:2726] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:44:14.325027 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:14.324537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:59523)
0810 14:44:14.324835 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:14.324902 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260810 14:44:14.974174 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:14.973704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:42543)
0810 14:44:14.974001 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:14.974071 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20260810 14:44:15.123939 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:15.123507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:60113)
0810 14:44:15.123776 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:15.123836 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260810 14:44:15.326440 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:15.325953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:45467)
0810 14:44:15.326250 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:15.326317 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
I20260810 14:44:15.766088 9945 raft_consensus.cc:482] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:44:15.766233 9945 raft_consensus.cc:504] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:44:15.766702 9945 leader_election.cc:288] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217), fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591)
W20260810 14:44:15.767516 9315 leader_election.cc:341] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217): Not found: Tablet not found: 62c607cc69ae421793ad6365fec185f3
W20260810 14:44:15.767643 9322 leader_election.cc:341] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591): Not found: Tablet not found: 62c607cc69ae421793ad6365fec185f3
I20260810 14:44:15.767740 9322 leader_election.cc:302] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b8b2e084cec486c8212030926e3e01e; no voters: f9dd5ae1fae6450a8ce49b16637ddde7, fb5125fb1b9941cc99710ee7bd6282f6
I20260810 14:44:15.767895 9945 raft_consensus.cc:2726] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:44:15.975420 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:15.974969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:50261)
0810 14:44:15.975252 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:15.975319 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":111,"threads_started":1}
I20260810 14:44:16.125255 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:16.124772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:49127)
0810 14:44:16.125042 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:16.125101 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
I20260810 14:44:16.327914 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:16.327425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:41555)
0810 14:44:16.327733 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:16.327796 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":90,"threads_started":1}
I20260810 14:44:16.976661 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:16.976169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:57459)
0810 14:44:16.976485 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:16.976557 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":86,"threads_started":1}
I20260810 14:44:17.126487 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:17.126043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:46819)
0810 14:44:17.126321 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:17.126388 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20260810 14:44:17.329190 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:17.328734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:50129)
0810 14:44:17.328992 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:17.329056 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
I20260810 14:44:17.978263 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:17.977723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:36723)
0810 14:44:17.978087 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:17.978160 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":104,"threads_started":1}
I20260810 14:44:18.127801 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:18.127350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:52425)
0810 14:44:18.127627 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:18.127698 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260810 14:44:18.330554 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:18.330104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:49037)
0810 14:44:18.330385 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:18.330451 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260810 14:44:18.530447 9957 raft_consensus.cc:482] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:44:18.530597 9957 raft_consensus.cc:504] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:44:18.531050 9957 leader_election.cc:288] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217), fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591)
W20260810 14:44:18.531944 9315 leader_election.cc:341] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217): Not found: Tablet not found: 62c607cc69ae421793ad6365fec185f3
W20260810 14:44:18.532079 9322 leader_election.cc:341] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591): Not found: Tablet not found: 62c607cc69ae421793ad6365fec185f3
I20260810 14:44:18.532168 9322 leader_election.cc:302] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b8b2e084cec486c8212030926e3e01e; no voters: f9dd5ae1fae6450a8ce49b16637ddde7, fb5125fb1b9941cc99710ee7bd6282f6
I20260810 14:44:18.532323 9957 raft_consensus.cc:2726] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:44:18.979600 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:18.979111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:43567)
0810 14:44:18.979426 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:18.979497 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":80,"threads_started":1}
I20260810 14:44:19.129228 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:19.128713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:49167)
0810 14:44:19.129016 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:19.129085 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":89,"threads_started":1}
I20260810 14:44:19.331796 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:19.331348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:55213)
0810 14:44:19.331635 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:19.331696 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
I20260810 14:44:19.981031 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:19.980561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:44281)
0810 14:44:19.980864 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:19.980929 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":84,"threads_started":1}
I20260810 14:44:20.130591 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:20.130075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:38473)
0810 14:44:20.130422 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:20.130489 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":81,"threads_started":1}
I20260810 14:44:20.333288 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:20.332716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:51329)
0810 14:44:20.333066 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:20.333135 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":104,"threads_started":1}
I20260810 14:44:20.811244 9966 raft_consensus.cc:482] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:44:20.811383 9966 raft_consensus.cc:504] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:44:20.811807 9966 leader_election.cc:288] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217), fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591)
W20260810 14:44:20.812654 9322 leader_election.cc:341] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591): Not found: Tablet not found: 62c607cc69ae421793ad6365fec185f3
W20260810 14:44:20.812774 9315 leader_election.cc:341] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217): Not found: Tablet not found: 62c607cc69ae421793ad6365fec185f3
I20260810 14:44:20.812839 9315 leader_election.cc:302] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b8b2e084cec486c8212030926e3e01e; no voters: f9dd5ae1fae6450a8ce49b16637ddde7, fb5125fb1b9941cc99710ee7bd6282f6
I20260810 14:44:20.812988 9966 raft_consensus.cc:2726] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:44:20.982401 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:20.981966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:45407)
0810 14:44:20.982232 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:20.982299 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260810 14:44:21.131810 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:21.131357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:56843)
0810 14:44:21.131637 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:21.131704 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260810 14:44:21.334636 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:21.334120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:38525)
0810 14:44:21.334432 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:21.334498 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":88,"threads_started":1}
I20260810 14:44:21.983656 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:21.983219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:40387)
0810 14:44:21.983486 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:21.983548 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
I20260810 14:44:22.133263 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:22.132690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:36755)
0810 14:44:22.133035 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:22.133106 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":95,"threads_started":1}
I20260810 14:44:22.335966 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:22.335501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:51775)
0810 14:44:22.335801 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:22.335867 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":91,"threads_started":1}
I20260810 14:44:22.985110 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:22.984680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:34219)
0810 14:44:22.984945 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:22.985010 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
I20260810 14:44:23.134743 9976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:23.134235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:46515)
0810 14:44:23.134566 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:23.134636 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
I20260810 14:44:23.337307 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:23.336785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:52853)
0810 14:44:23.337110 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:23.337204 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":86,"threads_started":1}
I20260810 14:44:23.986577 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:23.986089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:39513)
0810 14:44:23.986387 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:23.986455 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
I20260810 14:44:24.136230 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:24.135763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:60795)
0810 14:44:24.136056 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:24.136127 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":86,"threads_started":1}
I20260810 14:44:24.338687 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:24.338156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:51715)
0810 14:44:24.338500 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:24.338577 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":87,"threads_started":1}
I20260810 14:44:24.987979 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:24.987417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:46295)
0810 14:44:24.987778 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:24.987874 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":116,"threads_started":1}
I20260810 14:44:25.137488 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:25.137013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:51351)
0810 14:44:25.137324 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:25.137385 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":102,"threads_started":1}
I20260810 14:44:25.340054 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:25.339595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:43967)
0810 14:44:25.339877 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:25.339947 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
I20260810 14:44:25.989461 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:25.988896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:41319)
0810 14:44:25.989285 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:25.989356 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":88,"threads_started":1}
I20260810 14:44:26.139050 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:26.138509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:54341)
0810 14:44:26.138873 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:26.138947 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":100,"threads_started":1}
I20260810 14:44:26.341691 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:26.341119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:51005)
0810 14:44:26.341507 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:26.341577 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":107,"threads_started":1}
I20260810 14:44:26.990939 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:26.990379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:43351)
0810 14:44:26.990762 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:26.990835 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":91,"threads_started":1}
I20260810 14:44:27.140606 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:27.140052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:51927)
0810 14:44:27.140431 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:27.140503 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260810 14:44:27.343241 10000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:27.342728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:46165)
0810 14:44:27.343054 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:27.343137 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":89,"threads_started":1}
I20260810 14:44:27.992559 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:27.992063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.1:50419)
0810 14:44:27.992381 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:27.992455 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260810 14:44:28.073060 10003 raft_consensus.cc:482] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:44:28.073235 10003 raft_consensus.cc:504] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dd5ae1fae6450a8ce49b16637ddde7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 40217 } } peers { permanent_uuid: "fb5125fb1b9941cc99710ee7bd6282f6" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 38591 } } peers { permanent_uuid: "9b8b2e084cec486c8212030926e3e01e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 43907 } }
I20260810 14:44:28.073709 10003 leader_election.cc:288] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217), fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591)
W20260810 14:44:28.074558 9315 leader_election.cc:341] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9dd5ae1fae6450a8ce49b16637ddde7 (127.15.84.3:40217): Not found: Tablet not found: 62c607cc69ae421793ad6365fec185f3
W20260810 14:44:28.074718 9322 leader_election.cc:341] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fb5125fb1b9941cc99710ee7bd6282f6 (127.15.84.2:38591): Not found: Tablet not found: 62c607cc69ae421793ad6365fec185f3
I20260810 14:44:28.074810 9322 leader_election.cc:302] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b8b2e084cec486c8212030926e3e01e; no voters: f9dd5ae1fae6450a8ce49b16637ddde7, fb5125fb1b9941cc99710ee7bd6282f6
I20260810 14:44:28.074977 10003 raft_consensus.cc:2726] T 62c607cc69ae421793ad6365fec185f3 P 9b8b2e084cec486c8212030926e3e01e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:44:28.142280 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:28.141779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.2:41433)
0810 14:44:28.142097 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:28.142163 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":99,"threads_started":1}
I20260810 14:44:28.340291 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 9235
I20260810 14:44:28.352917 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:28.350816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40369 (local address 127.15.84.3:42731)
0810 14:44:28.352665 (+ 1849us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:28.352771 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.62:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1741,"thread_start_us":1186,"threads_started":1}
I20260810 14:44:28.363176 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 9505
I20260810 14:44:28.377542 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 9724
2026-08-10T14:44:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (60325 ms)
[----------] 30 tests from SecurityITest (249543 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:28 dist-test-slave-rmhg krb5kdc[10012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:28 dist-test-slave-rmhg krb5kdc[10012](info): set up 2 sockets
Aug 10 14:44:28 dist-test-slave-rmhg krb5kdc[10012](info): commencing operation
krb5kdc: starting...
W20260810 14:44:30.025449 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:30 dist-test-slave-rmhg krb5kdc[10012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:44:30.157131 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:34795
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:42313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:34795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:30.278340 10032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:30.278652 10032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:30.278713 10032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:30.282821 10032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:30.282899 10032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:30.282924 10032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:30.282943 10032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:30.282963 10032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:30.288652 10032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:42313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:34795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:34795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:30.289896 10032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:30.290979 10032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:44:30.300436 10032 server_base.cc:1034] running on GCE node
W20260810 14:44:30.300376 10042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:30.300376 10046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:30.300376 10048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:30.301348 10032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:30.302515 10032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:30.303694 10032 hybrid_clock.cc:648] HybridClock initialized: now 1786373070303673 us; error 34 us; skew 500 ppm
Aug 10 14:44:30 dist-test-slave-rmhg krb5kdc[10012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373070, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:30.306648 10032 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:44:30.307636 10032 webserver.cc:460] Webserver started at http://127.15.84.62:46249/ using document root <none> and password file <none>
I20260810 14:44:30.308257 10032 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:30.308326 10032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:30.308521 10032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:30.310734 10032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "d8d5e68d5f5f46c880d746c8b2ea8542"
format_stamp: "Formatted at 2026-08-10 14:44:30 on dist-test-slave-rmhg"
server_key: "a7d359969887cb2add939d4725917383"
server_key_iv: "442d577a6e7f5be3e82412d751d4d643"
server_key_version: "encryptionkey@0"
I20260810 14:44:30.311276 10032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "d8d5e68d5f5f46c880d746c8b2ea8542"
format_stamp: "Formatted at 2026-08-10 14:44:30 on dist-test-slave-rmhg"
server_key: "a7d359969887cb2add939d4725917383"
server_key_iv: "442d577a6e7f5be3e82412d751d4d643"
server_key_version: "encryptionkey@0"
I20260810 14:44:30.315513 10032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:44:30.318603 10058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:30.319936 10032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:44:30.320071 10032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "d8d5e68d5f5f46c880d746c8b2ea8542"
format_stamp: "Formatted at 2026-08-10 14:44:30 on dist-test-slave-rmhg"
server_key: "a7d359969887cb2add939d4725917383"
server_key_iv: "442d577a6e7f5be3e82412d751d4d643"
server_key_version: "encryptionkey@0"
I20260810 14:44:30.320195 10032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:30.336820 10032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:30.340252 10032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:30.340451 10032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:30.347981 10032 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:34795
I20260810 14:44:30.348052 10147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:34795 every 8 connection(s)
I20260810 14:44:30.349021 10032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:30.351964 10148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:30.355472 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 10032
I20260810 14:44:30.355616 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:30.355908 15696 external_mini_cluster.cc:1428] Setting key 8df973bcb2ade100f7b9b76d0fbb59a9
I20260810 14:44:30.358362 10148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542: Bootstrap starting.
I20260810 14:44:30.360970 10148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:30.361917 10148 log.cc:824] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:30.364379 10148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542: No bootstrap required, opened a new log
Aug 10 14:44:30 dist-test-slave-rmhg krb5kdc[10012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:44:30.367815 10148 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d5e68d5f5f46c880d746c8b2ea8542" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34795 } }
I20260810 14:44:30.368067 10148 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:30.368108 10148 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d5e68d5f5f46c880d746c8b2ea8542, State: Initialized, Role: FOLLOWER
I20260810 14:44:30.368599 10148 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d5e68d5f5f46c880d746c8b2ea8542" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34795 } }
I20260810 14:44:30.368752 10148 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:30.368821 10148 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:30.368937 10148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:30.370205 10148 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d5e68d5f5f46c880d746c8b2ea8542" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34795 } }
I20260810 14:44:30.370620 10148 leader_election.cc:302] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8d5e68d5f5f46c880d746c8b2ea8542; no voters:
I20260810 14:44:30.371025 10148 leader_election.cc:288] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:30.371115 10155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:30.371346 10155 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [term 1 LEADER]: Becoming Leader. State: Replica: d8d5e68d5f5f46c880d746c8b2ea8542, State: Running, Role: LEADER
I20260810 14:44:30.371768 10155 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d5e68d5f5f46c880d746c8b2ea8542" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34795 } }
I20260810 14:44:30.372032 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:30.357431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.15.84.62:34795)
0810 14:44:30.357858 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:30.357874 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:30.357912 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:44:30.358613 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:30.358633 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:30.358987 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:30.359297 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:30.360200 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:30.361363 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:30.362542 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:30.363171 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:30.363305 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:30.366137 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:30.366142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:30.366158 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:30.366192 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:30.368421 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:30.368944 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:30.368952 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:30.368959 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:30.369117 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:30.369424 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:30.369428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:30.369429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:30.370011 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:30.370256 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:30.370521 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 14:44:30.370898 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":107,"threads_started":1}
I20260810 14:44:30.372411 10148 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:30.373647 10156 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8d5e68d5f5f46c880d746c8b2ea8542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d5e68d5f5f46c880d746c8b2ea8542" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34795 } } }
I20260810 14:44:30.373643 10157 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8d5e68d5f5f46c880d746c8b2ea8542. Latest consensus state: current_term: 1 leader_uuid: "d8d5e68d5f5f46c880d746c8b2ea8542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d5e68d5f5f46c880d746c8b2ea8542" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34795 } } }
I20260810 14:44:30.373816 10157 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:30.373817 10156 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:30.374668 10168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:30.378170 10168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:30.384303 10168 catalog_manager.cc:1347] Generated new cluster ID: 5d2ddabe19eb45349f470540854ae0c4
I20260810 14:44:30.384389 10168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:30.419212 10168 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:30.420286 10168 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:30.430809 10168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542: Generated new TSK 0
I20260810 14:44:30.431576 10168 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:44:30.472337 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34795
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:42313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:30.612627 10186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:30.612926 10186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:30.612972 10186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:30.617195 10186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:30.617276 10186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:30.617339 10186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:44:30.622977 10186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:42313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:30.624183 10186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:30.625316 10186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:30.637509 10197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:30.637506 10194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:30.639060 10193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:30.639495 10186 server_base.cc:1034] running on GCE node
I20260810 14:44:30.640107 10186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:30.641026 10186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:30.642216 10186 hybrid_clock.cc:648] HybridClock initialized: now 1786373070642182 us; error 54 us; skew 500 ppm
Aug 10 14:44:30 dist-test-slave-rmhg krb5kdc[10012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373070, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:30.645248 10186 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:44:30.646243 10186 webserver.cc:460] Webserver started at http://127.15.84.1:40981/ using document root <none> and password file <none>
I20260810 14:44:30.646816 10186 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:30.646883 10186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:30.647081 10186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:30.649338 10186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b"
format_stamp: "Formatted at 2026-08-10 14:44:30 on dist-test-slave-rmhg"
server_key: "dab823ce2d44a2253c79301bb7f0c9e5"
server_key_iv: "79aee72cf8a8bb2c608d4dff737cb879"
server_key_version: "encryptionkey@0"
I20260810 14:44:30.649904 10186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b"
format_stamp: "Formatted at 2026-08-10 14:44:30 on dist-test-slave-rmhg"
server_key: "dab823ce2d44a2253c79301bb7f0c9e5"
server_key_iv: "79aee72cf8a8bb2c608d4dff737cb879"
server_key_version: "encryptionkey@0"
I20260810 14:44:30.654138 10186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:44:30.656759 10205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:30.658108 10186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:44:30.658231 10186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b"
format_stamp: "Formatted at 2026-08-10 14:44:30 on dist-test-slave-rmhg"
server_key: "dab823ce2d44a2253c79301bb7f0c9e5"
server_key_iv: "79aee72cf8a8bb2c608d4dff737cb879"
server_key_version: "encryptionkey@0"
I20260810 14:44:30.658342 10186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:30.670680 10186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:30.674361 10186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:30.674556 10186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:30.675209 10186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:30.676100 10186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:30.676164 10186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:30.676220 10186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:30.676245 10186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:30.685519 10186 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:41141
I20260810 14:44:30.685563 10365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:41141 every 8 connection(s)
I20260810 14:44:30.686511 10186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:44:30.691931 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 10186
I20260810 14:44:30.692052 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:44:30.692324 15696 external_mini_cluster.cc:1428] Setting key f09209e4076e880f16531a319ddae3cf
Aug 10 14:44:30 dist-test-slave-rmhg krb5kdc[10012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373070, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:44:30.702085 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:30.688432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:59783 (local address 127.15.84.62:34795)
0810 14:44:30.688593 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:30.688597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:30.689537 (+ 940us) server_negotiation.cc:407] Connection header received
0810 14:44:30.690872 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:30.690875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:30.690938 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:30.691020 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:30.692370 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:30.692976 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:30.694643 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:30.694944 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:30.695065 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:30.697986 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:30.697990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:30.697992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:30.698030 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:30.699616 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:30.700205 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:30.700209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:30.700212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:30.700312 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:30.700684 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:30.700688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:30.700690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:30.700950 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:30.701070 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:30.701736 (+ 666us) server_negotiation.cc:299] Negotiation successful
0810 14:44:30.701891 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:44:30.703015 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:30.688858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:34795 (local address 127.15.84.1:59783)
0810 14:44:30.689377 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:30.689424 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:30.690632 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:30.691227 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:30.691234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:30.691629 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:30.692208 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:30.692227 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:30.693205 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:30.693209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:30.694505 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:30.694518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:30.695137 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:30.695141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:30.695309 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:30.695929 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:44:30.695955 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:30.697796 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:30.699769 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:30.699775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:30.699791 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:30.700074 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:30.700415 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:30.700417 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:30.700419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:30.700579 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:30.701088 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:30.701093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:30.701443 (+ 350us) client_negotiation.cc:769] Sending connection context
0810 14:44:30.701626 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 14:44:30.701975 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
I20260810 14:44:30.704365 10368 heartbeater.cc:347] Connected to a master server at 127.15.84.62:34795
I20260810 14:44:30.704641 10368 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:30.705264 10368 heartbeater.cc:511] Master 127.15.84.62:34795 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
I20260810 14:44:30.707132 10088 ts_manager.cc:194] Registered new tserver with Master: 7a7bce1d7c2c4e4784540ab64bac0e3b (127.15.84.1:41141)
I20260810 14:44:30.708349 10088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:59783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:44:30.724144 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34795
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:42313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:30.843360 10378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:30.843657 10378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:30.843729 10378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:30.847916 10378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:30.847996 10378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:30.848090 10378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:44:30.853793 10378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:42313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:30.855077 10378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:30.856163 10378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:30.866791 10397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:30.866832 10389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:30.866801 10390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:30.867020 10378 server_base.cc:1034] running on GCE node
I20260810 14:44:30.867511 10378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:30.868208 10378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:30.869380 10378 hybrid_clock.cc:648] HybridClock initialized: now 1786373070869356 us; error 38 us; skew 500 ppm
Aug 10 14:44:30 dist-test-slave-rmhg krb5kdc[10012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373070, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:30.872421 10378 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:44:30.873595 10378 webserver.cc:460] Webserver started at http://127.15.84.2:33991/ using document root <none> and password file <none>
I20260810 14:44:30.874181 10378 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:30.874245 10378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:30.874495 10378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:30.876813 10378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "354155c4dd5242ddb8d3eb450d5db8c4"
format_stamp: "Formatted at 2026-08-10 14:44:30 on dist-test-slave-rmhg"
server_key: "1ba26e1b874c23f8ebced2c18988425a"
server_key_iv: "6f0aa22a3ad3be20451a5541784334da"
server_key_version: "encryptionkey@0"
I20260810 14:44:30.877444 10378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "354155c4dd5242ddb8d3eb450d5db8c4"
format_stamp: "Formatted at 2026-08-10 14:44:30 on dist-test-slave-rmhg"
server_key: "1ba26e1b874c23f8ebced2c18988425a"
server_key_iv: "6f0aa22a3ad3be20451a5541784334da"
server_key_version: "encryptionkey@0"
I20260810 14:44:30.881491 10378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:44:30.884117 10409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:30.885598 10378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:44:30.885731 10378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "354155c4dd5242ddb8d3eb450d5db8c4"
format_stamp: "Formatted at 2026-08-10 14:44:30 on dist-test-slave-rmhg"
server_key: "1ba26e1b874c23f8ebced2c18988425a"
server_key_iv: "6f0aa22a3ad3be20451a5541784334da"
server_key_version: "encryptionkey@0"
I20260810 14:44:30.885843 10378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:30.899421 10378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:30.902619 10378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:30.902818 10378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:30.903455 10378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:30.904349 10378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:30.904400 10378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:30.904456 10378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:30.904503 10378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:30.913519 10378 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:36765
I20260810 14:44:30.913563 10598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:36765 every 8 connection(s)
I20260810 14:44:30.914525 10378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:44:30.922206 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 10378
I20260810 14:44:30.922349 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:44:30.922688 15696 external_mini_cluster.cc:1428] Setting key 31884431ad6609d2c1e4f8eba3a26870
Aug 10 14:44:30 dist-test-slave-rmhg krb5kdc[10012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373070, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:44:30.929844 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:30.916678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:48715 (local address 127.15.84.62:34795)
0810 14:44:30.916885 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:30.916890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:30.918035 (+ 1145us) server_negotiation.cc:407] Connection header received
0810 14:44:30.919314 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:30.919317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:30.919371 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:30.919455 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:30.920757 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:30.921378 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:30.922646 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:30.922975 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:30.923120 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:30.925938 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:30.925942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:30.925946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:30.925978 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:30.927358 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:30.928021 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:30.928023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:30.928025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:30.928114 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:30.928484 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:30.928486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:30.928487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:30.928675 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:30.928749 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:30.929564 (+ 815us) server_negotiation.cc:299] Negotiation successful
0810 14:44:30.929707 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 14:44:30.931195 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:30.917344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:34795 (local address 127.15.84.2:48715)
0810 14:44:30.917859 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:30.917908 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:30.919089 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:30.919617 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:30.919624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:30.920044 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:30.920602 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:30.920617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:30.921512 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:30.921515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:30.922516 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:30.922528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:30.923220 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:30.923226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:30.923430 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:30.924085 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:44:30.924108 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:30.925750 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:30.927489 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:30.927497 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:30.927518 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:30.927925 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:30.928194 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:30.928197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:30.928198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:30.928400 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:30.928774 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:30.928781 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:30.929197 (+ 416us) client_negotiation.cc:769] Sending connection context
0810 14:44:30.929454 (+ 257us) client_negotiation.cc:240] Negotiation successful
0810 14:44:30.929921 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":145,"threads_started":1}
I20260810 14:44:30.932605 10601 heartbeater.cc:347] Connected to a master server at 127.15.84.62:34795
I20260810 14:44:30.932917 10601 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:30.933646 10601 heartbeater.cc:511] Master 127.15.84.62:34795 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
I20260810 14:44:30.935178 10088 ts_manager.cc:194] Registered new tserver with Master: 354155c4dd5242ddb8d3eb450d5db8c4 (127.15.84.2:36765)
I20260810 14:44:30.935806 10088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:48715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:44:30.956269 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34795
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:42313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:31.077525 10609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:31.077821 10609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:31.077867 10609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:31.082121 10609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:31.082212 10609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:31.082278 10609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:44:31.087863 10609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:42313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:31.089053 10609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:31.090191 10609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:31.100325 10621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:31.100325 10624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:31.100328 10618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:31.100616 10609 server_base.cc:1034] running on GCE node
I20260810 14:44:31.101121 10609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:31.101837 10609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:31.103010 10609 hybrid_clock.cc:648] HybridClock initialized: now 1786373071102985 us; error 36 us; skew 500 ppm
Aug 10 14:44:31 dist-test-slave-rmhg krb5kdc[10012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373071, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:31.106060 10609 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:44:31.107030 10609 webserver.cc:460] Webserver started at http://127.15.84.3:43633/ using document root <none> and password file <none>
I20260810 14:44:31.107616 10609 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:31.107712 10609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:31.107926 10609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:31.110231 10609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "b800bb83674c4a89a7babfe334bafac3"
format_stamp: "Formatted at 2026-08-10 14:44:31 on dist-test-slave-rmhg"
server_key: "6beb9e3dc85dda69a8d7641a90a9b9f5"
server_key_iv: "eb226d9772469ab11850e5b023ea6dfa"
server_key_version: "encryptionkey@0"
I20260810 14:44:31.110821 10609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "b800bb83674c4a89a7babfe334bafac3"
format_stamp: "Formatted at 2026-08-10 14:44:31 on dist-test-slave-rmhg"
server_key: "6beb9e3dc85dda69a8d7641a90a9b9f5"
server_key_iv: "eb226d9772469ab11850e5b023ea6dfa"
server_key_version: "encryptionkey@0"
I20260810 14:44:31.114908 10609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:44:31.117635 10634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:31.118978 10609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:44:31.119107 10609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "b800bb83674c4a89a7babfe334bafac3"
format_stamp: "Formatted at 2026-08-10 14:44:31 on dist-test-slave-rmhg"
server_key: "6beb9e3dc85dda69a8d7641a90a9b9f5"
server_key_iv: "eb226d9772469ab11850e5b023ea6dfa"
server_key_version: "encryptionkey@0"
I20260810 14:44:31.119217 10609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:31.137522 10609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:31.140811 10609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:31.141010 10609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:31.141717 10609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:31.142627 10609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:31.142689 10609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:31.142746 10609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:31.142772 10609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:31.151779 10609 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:33555
I20260810 14:44:31.151852 10810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:33555 every 8 connection(s)
I20260810 14:44:31.152782 10609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:44:31.154965 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 10609
I20260810 14:44:31.155123 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:44:31.155400 15696 external_mini_cluster.cc:1428] Setting key 41c1b417e277f04382fd4e30ba8393df
Aug 10 14:44:31 dist-test-slave-rmhg krb5kdc[10012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373071, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:44:31.167701 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.154875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:41001 (local address 127.15.84.62:34795)
0810 14:44:31.155039 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:31.155044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:31.156072 (+ 1028us) server_negotiation.cc:407] Connection header received
0810 14:44:31.157574 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:31.157577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:31.157638 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:31.157713 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:31.159000 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.159549 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.160762 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.160989 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.161104 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.163795 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:31.163799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:31.163801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:31.163833 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:31.165408 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:31.166016 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:31.166019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:31.166021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:31.166113 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:31.166544 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:31.166548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:31.166549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:31.166766 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:31.166852 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:31.167432 (+ 580us) server_negotiation.cc:299] Negotiation successful
0810 14:44:31.167569 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:44:31.168629 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.155346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:34795 (local address 127.15.84.3:41001)
0810 14:44:31.155903 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:31.155953 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:31.157310 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:31.157929 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:31.157937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:31.158334 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:31.158853 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:31.158867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.159678 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.159681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:31.160634 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:31.160644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.161146 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.161175 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:31.161358 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.161966 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:44:31.161984 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:31.163558 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:31.165584 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:31.165591 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:31.165607 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:31.165906 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:31.166222 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:31.166225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:31.166227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:31.166423 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:31.166853 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:31.166859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:31.167158 (+ 299us) client_negotiation.cc:769] Sending connection context
0810 14:44:31.167337 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 14:44:31.167634 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":128,"threads_started":1}
I20260810 14:44:31.170037 10812 heartbeater.cc:347] Connected to a master server at 127.15.84.62:34795
I20260810 14:44:31.170282 10812 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:31.170795 10812 heartbeater.cc:511] Master 127.15.84.62:34795 requested a full tablet report, sending...
I20260810 14:44:31.171830 10084 ts_manager.cc:194] Registered new tserver with Master: b800bb83674c4a89a7babfe334bafac3 (127.15.84.3:33555)
I20260810 14:44:31.172457 10084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:41001
I20260810 14:44:31.182238 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:44:31.192592 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.184399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48978 (local address 127.15.84.62:34795)
0810 14:44:31.184578 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:31.184582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:31.184790 (+ 208us) server_negotiation.cc:407] Connection header received
0810 14:44:31.184951 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:31.184954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:31.185002 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:31.185065 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:31.185920 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.186474 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.187374 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.187676 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.187764 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.189281 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:31.189285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:31.189287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:31.189316 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:31.190857 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:31.191305 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:31.191309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:31.191311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:31.191415 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:31.191674 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:31.191677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:31.191678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:31.191939 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:31.192066 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:31.192310 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 14:44:31.192424 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:44:31.196179 10084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:44:31.199478 10084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:44:31.215646 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.209641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.15.84.2:36765)
0810 14:44:31.210396 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:31.210410 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:31.210559 (+ 149us) server_negotiation.cc:407] Connection header received
0810 14:44:31.210798 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:31.210804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:31.211056 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:31.211246 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:31.211547 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.212499 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.214091 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.214855 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.214973 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.215163 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:31.215352 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 14:44:31.215472 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":145,"threads_started":1}
I20260810 14:44:31.215642 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.209411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:36765 (local address 127.0.0.1:42840)
0810 14:44:31.210411 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:31.210472 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:31.210629 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:31.211225 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:31.211231 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:31.211253 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:31.211414 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:31.211422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.212633 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.212636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:31.213959 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:31.213971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.214986 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.214990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:31.215118 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.215165 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 14:44:31.215299 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 14:44:31.215453 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":1,"thread_start_us":112,"threads_started":1}
I20260810 14:44:31.216105 10832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.209866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:41141 (local address 127.0.0.1:44134)
0810 14:44:31.210605 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:31.210619 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:31.211045 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:31.211779 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:31.211782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:31.211799 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:31.211871 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:31.211876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.213285 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.213289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:31.214528 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:31.214535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.215706 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.215710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:31.215814 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.215839 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:44:31.215906 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 14:44:31.215975 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":62,"thread_start_us":91,"threads_started":1}
I20260810 14:44:31.216475 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.210639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.15.84.3:33555)
0810 14:44:31.211086 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:31.211097 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:31.211117 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:44:31.211192 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:31.211197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:31.211387 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:31.211531 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:31.211755 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.212879 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.214492 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.215341 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.215448 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.215600 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:31.216211 (+ 611us) server_negotiation.cc:299] Negotiation successful
0810 14:44:31.216320 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":104,"threads_started":1}
I20260810 14:44:31.216917 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.209931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44134 (local address 127.15.84.1:41141)
0810 14:44:31.210946 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:31.210955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:31.210980 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:44:31.211208 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:31.211212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:31.211387 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:31.211531 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:31.211992 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.213102 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.214644 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.215373 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.215470 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.215626 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:31.216384 (+ 758us) server_negotiation.cc:299] Negotiation successful
0810 14:44:31.216500 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":69,"threads_started":1}
I20260810 14:44:31.216349 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.210239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:33555 (local address 127.0.0.1:57584)
0810 14:44:31.210978 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:31.210992 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:31.211070 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:31.211528 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:31.211531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:31.211550 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:31.211660 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:31.211667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.213039 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.213042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:31.214345 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:31.214356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.215688 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.215691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:31.216052 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.216073 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 14:44:31.216140 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 14:44:31.216219 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":29,"thread_start_us":89,"threads_started":1}
I20260810 14:44:31.219527 10480 tablet_service.cc:1467] Processing CreateTablet for tablet e807b120fc274c49a572b9ab1d4fd267 (DEFAULT_TABLE table=test-table [id=4f9c5127774c40f2a09e2e25a340e9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:31.219862 10277 tablet_service.cc:1467] Processing CreateTablet for tablet e807b120fc274c49a572b9ab1d4fd267 (DEFAULT_TABLE table=test-table [id=4f9c5127774c40f2a09e2e25a340e9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:31.220376 10713 tablet_service.cc:1467] Processing CreateTablet for tablet e807b120fc274c49a572b9ab1d4fd267 (DEFAULT_TABLE table=test-table [id=4f9c5127774c40f2a09e2e25a340e9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:31.220909 10480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e807b120fc274c49a572b9ab1d4fd267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:31.220909 10277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e807b120fc274c49a572b9ab1d4fd267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:31.221549 10713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e807b120fc274c49a572b9ab1d4fd267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:31.228207 10834 tablet_bootstrap.cc:492] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3: Bootstrap starting.
I20260810 14:44:31.229892 10835 tablet_bootstrap.cc:492] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b: Bootstrap starting.
I20260810 14:44:31.229892 10836 tablet_bootstrap.cc:492] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4: Bootstrap starting.
I20260810 14:44:31.231017 10834 tablet_bootstrap.cc:654] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:31.232177 10834 log.cc:824] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:31.232309 10835 tablet_bootstrap.cc:654] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:31.232692 10836 tablet_bootstrap.cc:654] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:31.233245 10835 log.cc:824] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:31.233542 10836 log.cc:824] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:31.234527 10834 tablet_bootstrap.cc:492] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3: No bootstrap required, opened a new log
I20260810 14:44:31.234762 10834 ts_tablet_manager.cc:1397] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 14:44:31.235438 10835 tablet_bootstrap.cc:492] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b: No bootstrap required, opened a new log
I20260810 14:44:31.235646 10835 ts_tablet_manager.cc:1397] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:44:31.235896 10836 tablet_bootstrap.cc:492] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4: No bootstrap required, opened a new log
I20260810 14:44:31.236083 10836 ts_tablet_manager.cc:1397] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:44:31.239017 10834 raft_consensus.cc:348] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36765 } } peers { permanent_uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41141 } } peers { permanent_uuid: "b800bb83674c4a89a7babfe334bafac3" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 33555 } }
I20260810 14:44:31.239393 10834 raft_consensus.cc:374] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:31.239248 10835 raft_consensus.cc:348] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36765 } } peers { permanent_uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41141 } } peers { permanent_uuid: "b800bb83674c4a89a7babfe334bafac3" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 33555 } }
I20260810 14:44:31.239459 10834 raft_consensus.cc:729] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b800bb83674c4a89a7babfe334bafac3, State: Initialized, Role: FOLLOWER
I20260810 14:44:31.239491 10835 raft_consensus.cc:374] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:31.239537 10835 raft_consensus.cc:729] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7bce1d7c2c4e4784540ab64bac0e3b, State: Initialized, Role: FOLLOWER
I20260810 14:44:31.240018 10835 consensus_queue.cc:260] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36765 } } peers { permanent_uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41141 } } peers { permanent_uuid: "b800bb83674c4a89a7babfe334bafac3" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 33555 } }
I20260810 14:44:31.240046 10834 consensus_queue.cc:260] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36765 } } peers { permanent_uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41141 } } peers { permanent_uuid: "b800bb83674c4a89a7babfe334bafac3" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 33555 } }
I20260810 14:44:31.240020 10836 raft_consensus.cc:348] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36765 } } peers { permanent_uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41141 } } peers { permanent_uuid: "b800bb83674c4a89a7babfe334bafac3" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 33555 } }
I20260810 14:44:31.240309 10836 raft_consensus.cc:374] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:31.240375 10836 raft_consensus.cc:729] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354155c4dd5242ddb8d3eb450d5db8c4, State: Initialized, Role: FOLLOWER
I20260810 14:44:31.240864 10836 consensus_queue.cc:260] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36765 } } peers { permanent_uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41141 } } peers { permanent_uuid: "b800bb83674c4a89a7babfe334bafac3" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 33555 } }
I20260810 14:44:31.241222 10834 ts_tablet_manager.cc:1428] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 14:44:31.241223 10835 ts_tablet_manager.cc:1428] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 14:44:31.241595 10812 heartbeater.cc:503] Master 127.15.84.62:34795 was elected leader, sending a full tablet report...
I20260810 14:44:31.241595 10368 heartbeater.cc:503] Master 127.15.84.62:34795 was elected leader, sending a full tablet report...
I20260810 14:44:31.242941 10601 heartbeater.cc:503] Master 127.15.84.62:34795 was elected leader, sending a full tablet report...
I20260810 14:44:31.242949 10836 ts_tablet_manager.cc:1428] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260810 14:44:31.279247 10849 raft_consensus.cc:482] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:44:31.279495 10849 raft_consensus.cc:504] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36765 } } peers { permanent_uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41141 } } peers { permanent_uuid: "b800bb83674c4a89a7babfe334bafac3" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 33555 } }
I20260810 14:44:31.281412 10849 leader_election.cc:288] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a7bce1d7c2c4e4784540ab64bac0e3b (127.15.84.1:41141), b800bb83674c4a89a7babfe334bafac3 (127.15.84.3:33555)
I20260810 14:44:31.285050 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.281076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:33555 (local address 127.15.84.2:50229)
0810 14:44:31.281235 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:31.281251 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:31.281348 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:31.281745 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:31.281749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:31.281770 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:31.281896 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:31.281904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.282879 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.282882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:31.283880 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:31.283892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.284601 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.284605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:31.284721 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.284749 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 14:44:31.284823 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 14:44:31.284900 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":49}
I20260810 14:44:31.285068 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.281353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:50229 (local address 127.15.84.3:33555)
0810 14:44:31.281478 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:31.281482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:31.281497 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:44:31.281554 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:31.281557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:31.281609 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:31.281693 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:31.282024 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.282768 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.284007 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.284451 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.284567 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.284629 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:31.284853 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 14:44:31.284937 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:44:31.285235 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.281229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:56811 (local address 127.15.84.1:41141)
0810 14:44:31.281356 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:31.281361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:31.281820 (+ 459us) server_negotiation.cc:407] Connection header received
0810 14:44:31.281944 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:31.281947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:31.282006 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:31.282068 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:31.282378 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.282957 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.284126 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.284601 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.284743 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.284805 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:31.285017 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 14:44:31.285105 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:44:31.285255 10851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.281090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:41141 (local address 127.15.84.2:56811)
0810 14:44:31.281720 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:31.281734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:31.281839 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:31.282107 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:31.282110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:31.282129 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:31.282268 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:31.282274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.283060 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.283064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:31.283993 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:31.284005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.284762 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.284765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:31.284870 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.284896 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:44:31.284977 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 14:44:31.285056 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":147,"thread_start_us":139,"threads_started":1}
I20260810 14:44:31.286074 10302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e807b120fc274c49a572b9ab1d4fd267" candidate_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b" is_pre_election: true
I20260810 14:44:31.286082 10742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e807b120fc274c49a572b9ab1d4fd267" candidate_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b800bb83674c4a89a7babfe334bafac3" is_pre_election: true
I20260810 14:44:31.286389 10302 raft_consensus.cc:2454] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 354155c4dd5242ddb8d3eb450d5db8c4 in term 0.
I20260810 14:44:31.286389 10742 raft_consensus.cc:2454] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 354155c4dd5242ddb8d3eb450d5db8c4 in term 0.
I20260810 14:44:31.286890 10413 leader_election.cc:302] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 354155c4dd5242ddb8d3eb450d5db8c4, b800bb83674c4a89a7babfe334bafac3; no voters:
I20260810 14:44:31.287178 10849 raft_consensus.cc:2781] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:44:31.287278 10849 raft_consensus.cc:482] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:44:31.287328 10849 raft_consensus.cc:3037] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:31.288434 10849 raft_consensus.cc:504] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36765 } } peers { permanent_uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41141 } } peers { permanent_uuid: "b800bb83674c4a89a7babfe334bafac3" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 33555 } }
I20260810 14:44:31.288807 10849 leader_election.cc:288] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [CANDIDATE]: Term 1 election: Requested vote from peers 7a7bce1d7c2c4e4784540ab64bac0e3b (127.15.84.1:41141), b800bb83674c4a89a7babfe334bafac3 (127.15.84.3:33555)
I20260810 14:44:31.289311 10302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e807b120fc274c49a572b9ab1d4fd267" candidate_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b"
I20260810 14:44:31.289310 10742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e807b120fc274c49a572b9ab1d4fd267" candidate_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b800bb83674c4a89a7babfe334bafac3"
I20260810 14:44:31.289461 10742 raft_consensus.cc:3037] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:31.289479 10302 raft_consensus.cc:3037] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:31.290885 10302 raft_consensus.cc:2454] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 354155c4dd5242ddb8d3eb450d5db8c4 in term 1.
I20260810 14:44:31.290885 10742 raft_consensus.cc:2454] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 354155c4dd5242ddb8d3eb450d5db8c4 in term 1.
I20260810 14:44:31.291282 10414 leader_election.cc:302] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 354155c4dd5242ddb8d3eb450d5db8c4, 7a7bce1d7c2c4e4784540ab64bac0e3b; no voters:
I20260810 14:44:31.291538 10849 raft_consensus.cc:2781] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:31.291805 10849 raft_consensus.cc:686] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [term 1 LEADER]: Becoming Leader. State: Replica: 354155c4dd5242ddb8d3eb450d5db8c4, State: Running, Role: LEADER
I20260810 14:44:31.292223 10849 consensus_queue.cc:237] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36765 } } peers { permanent_uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41141 } } peers { permanent_uuid: "b800bb83674c4a89a7babfe334bafac3" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 33555 } }
I20260810 14:44:31.295737 10084 catalog_manager.cc:5495] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 354155c4dd5242ddb8d3eb450d5db8c4 (127.15.84.2). New cstate: current_term: 1 leader_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354155c4dd5242ddb8d3eb450d5db8c4" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b800bb83674c4a89a7babfe334bafac3" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 33555 } health_report { overall_health: UNKNOWN } } }
I20260810 14:44:31.327011 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.323085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.15.84.2:36765)
0810 14:44:31.323216 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:31.323219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:31.323235 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:44:31.323332 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:31.323336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:31.323410 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:31.323512 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:44:31.323737 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.324390 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.325442 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.325677 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.325762 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.326054 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:44:31.326524 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:44:31.326632 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:31.326794 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 14:44:31.326860 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:44:31.333896 10742 raft_consensus.cc:1261] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3 [term 1 FOLLOWER]: Refusing update from remote peer 354155c4dd5242ddb8d3eb450d5db8c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:44:31.333896 10302 raft_consensus.cc:1261] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b [term 1 FOLLOWER]: Refusing update from remote peer 354155c4dd5242ddb8d3eb450d5db8c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:44:31.334757 10852 consensus_queue.cc:1035] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b800bb83674c4a89a7babfe334bafac3" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:44:31.334972 10849 consensus_queue.cc:1035] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7bce1d7c2c4e4784540ab64bac0e3b" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 41141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:44:31.354404 10088 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:44:31.354630 10088 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:44:31.357652 10088 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8d5e68d5f5f46c880d746c8b2ea8542: Sending DeleteTablet for 3 replicas of tablet e807b120fc274c49a572b9ab1d4fd267
I20260810 14:44:31.358613 10480 tablet_service.cc:1514] Processing DeleteTablet for tablet e807b120fc274c49a572b9ab1d4fd267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:31 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:42840
I20260810 14:44:31.358916 10277 tablet_service.cc:1514] Processing DeleteTablet for tablet e807b120fc274c49a572b9ab1d4fd267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:31 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:44134
I20260810 14:44:31.359498 10866 tablet_replica.cc:330] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b: stopping tablet replica
I20260810 14:44:31.359869 10866 raft_consensus.cc:2229] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:44:31.360146 10713 tablet_service.cc:1514] Processing DeleteTablet for tablet e807b120fc274c49a572b9ab1d4fd267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:31 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:57584
I20260810 14:44:31.360253 10866 raft_consensus.cc:2258] T e807b120fc274c49a572b9ab1d4fd267 P 7a7bce1d7c2c4e4784540ab64bac0e3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:44:31.361236 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 10186
I20260810 14:44:31.371935 10867 tablet_replica.cc:330] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3: stopping tablet replica
I20260810 14:44:31.372426 10867 raft_consensus.cc:2229] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:44:31.372875 10867 raft_consensus.cc:2258] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:44:31.373512 10865 tablet_replica.cc:330] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4: stopping tablet replica
I20260810 14:44:31.373911 10865 raft_consensus.cc:2229] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:44:31.374428 10865 raft_consensus.cc:2258] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:44:31.375415 10867 ts_tablet_manager.cc:1905] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:44:31.376080 10865 ts_tablet_manager.cc:1905] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:44:31.378737 10867 ts_tablet_manager.cc:1918] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:44:31.378855 10865 ts_tablet_manager.cc:1918] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:44:31.378962 10865 log.cc:1196] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/e807b120fc274c49a572b9ab1d4fd267
I20260810 14:44:31.379311 10865 ts_tablet_manager.cc:1939] T e807b120fc274c49a572b9ab1d4fd267 P 354155c4dd5242ddb8d3eb450d5db8c4: Deleting consensus metadata
W20260810 14:44:31.379451 10066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:41141 (error 108)
I20260810 14:44:31.379523 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 10378
I20260810 14:44:31.389757 10867 log.cc:1196] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/e807b120fc274c49a572b9ab1d4fd267
I20260810 14:44:31.390259 10867 ts_tablet_manager.cc:1939] T e807b120fc274c49a572b9ab1d4fd267 P b800bb83674c4a89a7babfe334bafac3: Deleting consensus metadata
I20260810 14:44:31.391839 10065 catalog_manager.cc:4841] TS b800bb83674c4a89a7babfe334bafac3 (127.15.84.3:33555): tablet e807b120fc274c49a572b9ab1d4fd267 (table test-table [id=4f9c5127774c40f2a09e2e25a340e9c5]) successfully deleted
I20260810 14:44:31.392109 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.391711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:41141 (local address 127.0.0.1:44148)
0810 14:44:31.391926 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:31.392018 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:41141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260810 14:44:31.392285 10066 catalog_manager.cc:4584] TS 7a7bce1d7c2c4e4784540ab64bac0e3b (127.15.84.1:41141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e807b120fc274c49a572b9ab1d4fd267: Network error: Client connection negotiation failed: client connection to 127.15.84.1:41141: connect: Connection refused (error 111)
I20260810 14:44:31.396817 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 10609
I20260810 14:44:31.399673 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.397015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:36765 (local address 127.0.0.1:42852)
0810 14:44:31.399488 (+ 2473us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:31.399560 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.2:36765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2356}
I20260810 14:44:31.423333 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 10032
2026-08-10T14:44:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2567 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T14:44:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:31Z Disabled control of system clock
I20260810 14:44:31.468724 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:44703
--webserver_interface=127.15.84.62
--webserver_port=0
--builtin_ntp_servers=127.15.84.20:40285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:44703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:44:31.589429 10873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:31.589749 10873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:31.589788 10873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:31.594000 10873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:31.594084 10873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:31.594097 10873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:31.594117 10873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:31.594131 10873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:31.599241 10873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:40285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:44703
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:44703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:31.600481 10873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:31.601965 10873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:31.610845 10880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:31.610895 10873 server_base.cc:1034] running on GCE node
W20260810 14:44:31.611101 10883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:31.611128 10881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:31.611780 10873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:31.613044 10873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:31.614223 10873 hybrid_clock.cc:648] HybridClock initialized: now 1786373071614202 us; error 37 us; skew 500 ppm
I20260810 14:44:31.616524 10873 webserver.cc:460] Webserver started at http://127.15.84.62:36013/ using document root <none> and password file <none>
I20260810 14:44:31.617268 10873 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:31.617362 10873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:31.617614 10873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:31.619889 10873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "5adbe28f834f44f387e8258f534f562a"
format_stamp: "Formatted at 2026-08-10 14:44:31 on dist-test-slave-rmhg"
server_key: "30f93972a3f8b704170b0ba3f6b8b571"
server_key_iv: "5808702ec79d5378bd6f811ccf22992f"
server_key_version: "encryptionkey@0"
I20260810 14:44:31.620507 10873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "5adbe28f834f44f387e8258f534f562a"
format_stamp: "Formatted at 2026-08-10 14:44:31 on dist-test-slave-rmhg"
server_key: "30f93972a3f8b704170b0ba3f6b8b571"
server_key_iv: "5808702ec79d5378bd6f811ccf22992f"
server_key_version: "encryptionkey@0"
I20260810 14:44:31.624718 10873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:44:31.627772 10892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:31.629269 10873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:44:31.629410 10873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "5adbe28f834f44f387e8258f534f562a"
format_stamp: "Formatted at 2026-08-10 14:44:31 on dist-test-slave-rmhg"
server_key: "30f93972a3f8b704170b0ba3f6b8b571"
server_key_iv: "5808702ec79d5378bd6f811ccf22992f"
server_key_version: "encryptionkey@0"
I20260810 14:44:31.629528 10873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:31.641852 10873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:31.642639 10873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:31.642854 10873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:31.650370 10873 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:44703
I20260810 14:44:31.650390 10978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:44703 every 8 connection(s)
I20260810 14:44:31.651445 10873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:31.654947 10979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:31.655985 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 10873
I20260810 14:44:31.656126 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:31.656451 15696 external_mini_cluster.cc:1428] Setting key 1ad3135889d29d2e3d212189dc929f5b
I20260810 14:44:31.661554 10979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a: Bootstrap starting.
I20260810 14:44:31.663956 10979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:31.664759 10979 log.cc:824] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:31.666049 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.658069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.15.84.62:44703)
0810 14:44:31.658804 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:31.658822 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:31.658860 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:44:31.659516 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:31.659534 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:31.659951 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:31.660279 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:31.660729 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.661767 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.662642 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.663131 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.663289 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.664143 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:31.664148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:31.664162 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:31.664578 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:31.664638 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:31.664659 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:31.664745 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:31.664885 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 14:44:31.665196 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":146,"threads_started":1}
I20260810 14:44:31.666854 10979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a: No bootstrap required, opened a new log
I20260810 14:44:31.670624 10979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5adbe28f834f44f387e8258f534f562a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44703 } }
I20260810 14:44:31.670887 10979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:31.670949 10979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5adbe28f834f44f387e8258f534f562a, State: Initialized, Role: FOLLOWER
I20260810 14:44:31.671377 10979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5adbe28f834f44f387e8258f534f562a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44703 } }
I20260810 14:44:31.671509 10979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:31.671576 10979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:31.671690 10979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:31.672709 10979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5adbe28f834f44f387e8258f534f562a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44703 } }
I20260810 14:44:31.673049 10979 leader_election.cc:302] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5adbe28f834f44f387e8258f534f562a; no voters:
I20260810 14:44:31.673389 10979 leader_election.cc:288] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:31.673496 10988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:31.673731 10988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [term 1 LEADER]: Becoming Leader. State: Replica: 5adbe28f834f44f387e8258f534f562a, State: Running, Role: LEADER
I20260810 14:44:31.674052 10988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5adbe28f834f44f387e8258f534f562a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44703 } }
I20260810 14:44:31.674779 10979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:31.675860 10994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5adbe28f834f44f387e8258f534f562a. Latest consensus state: current_term: 1 leader_uuid: "5adbe28f834f44f387e8258f534f562a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5adbe28f834f44f387e8258f534f562a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44703 } } }
I20260810 14:44:31.675868 10990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5adbe28f834f44f387e8258f534f562a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5adbe28f834f44f387e8258f534f562a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44703 } } }
I20260810 14:44:31.676009 10990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:31.676009 10994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:31.676424 11002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:31.679260 11002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:31.684926 11002 catalog_manager.cc:1347] Generated new cluster ID: 69dda14b91f54f99b6157949b00f75e9
I20260810 14:44:31.685005 11002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:31.699570 11002 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:31.700646 11002 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:31.705865 11002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a: Generated new TSK 0
I20260810 14:44:31.706810 11002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:31.710976 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:44703
--builtin_ntp_servers=127.15.84.20:40285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:44:31.831650 11015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:31.831955 11015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:31.831995 11015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:31.836246 11015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:31.836325 11015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:31.836386 11015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:44:31.841504 11015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:40285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:44703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:31.842666 11015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:31.844077 11015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:31.854367 11021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:31.854384 11026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:31.854341 11022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:31.854734 11015 server_base.cc:1034] running on GCE node
I20260810 14:44:31.855223 11015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:31.855934 11015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:31.857110 11015 hybrid_clock.cc:648] HybridClock initialized: now 1786373071857084 us; error 40 us; skew 500 ppm
I20260810 14:44:31.859201 11015 webserver.cc:460] Webserver started at http://127.15.84.1:37741/ using document root <none> and password file <none>
I20260810 14:44:31.859826 11015 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:31.859894 11015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:31.860121 11015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:31.862429 11015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "20149bfd33d44d1e96c59cef2c5a9b3c"
format_stamp: "Formatted at 2026-08-10 14:44:31 on dist-test-slave-rmhg"
server_key: "d1c3b420b70175a90595a0b8e131022d"
server_key_iv: "694eb7b4f4f3c106f795312bc56b6d94"
server_key_version: "encryptionkey@0"
I20260810 14:44:31.863034 11015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "20149bfd33d44d1e96c59cef2c5a9b3c"
format_stamp: "Formatted at 2026-08-10 14:44:31 on dist-test-slave-rmhg"
server_key: "d1c3b420b70175a90595a0b8e131022d"
server_key_iv: "694eb7b4f4f3c106f795312bc56b6d94"
server_key_version: "encryptionkey@0"
I20260810 14:44:31.867107 11015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 14:44:31.869922 11036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:31.871394 11015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:44:31.871531 11015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "20149bfd33d44d1e96c59cef2c5a9b3c"
format_stamp: "Formatted at 2026-08-10 14:44:31 on dist-test-slave-rmhg"
server_key: "d1c3b420b70175a90595a0b8e131022d"
server_key_iv: "694eb7b4f4f3c106f795312bc56b6d94"
server_key_version: "encryptionkey@0"
I20260810 14:44:31.871666 11015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:31.896858 11015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:31.897675 11015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:31.897864 11015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:31.898520 11015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:31.899518 11015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:31.899569 11015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:31.899624 11015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:31.899653 11015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:31.908458 11015 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:45939
I20260810 14:44:31.908488 11206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:45939 every 8 connection(s)
I20260810 14:44:31.909456 11015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:44:31.919303 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 11015
I20260810 14:44:31.919441 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:44:31.919770 15696 external_mini_cluster.cc:1428] Setting key fbe99e0a9d2b5f832fbf8a92cb1b2807
I20260810 14:44:31.919792 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.911503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:46135 (local address 127.15.84.62:44703)
0810 14:44:31.911679 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:31.911683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:31.912987 (+ 1304us) server_negotiation.cc:407] Connection header received
0810 14:44:31.914467 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:31.914470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:31.914537 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:31.914632 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:31.915702 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.916296 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.917642 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.917894 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.918022 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.919252 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:31.919255 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:31.919257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:31.919272 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:31.919291 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:31.919299 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:31.919358 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:31.919541 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 14:44:31.919640 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:44:31.920776 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:31.912058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44703 (local address 127.15.84.1:46135)
0810 14:44:31.912813 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:31.912859 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:31.914218 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:31.914856 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:31.914865 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:31.914972 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:31.915529 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:31.915545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.916422 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.916425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:31.917502 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:31.917513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:31.918126 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:31.918130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:31.918307 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:31.918931 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:44:31.918932 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:31.918988 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:44:31.919084 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:31.919371 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:31.919381 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:31.919388 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:44:31.919510 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 14:44:31.919793 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":126,"threads_started":1}
I20260810 14:44:31.921931 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:44703
--builtin_ntp_servers=127.15.84.20:40285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:44:31.922178 11207 heartbeater.cc:347] Connected to a master server at 127.15.84.62:44703
I20260810 14:44:31.922463 11207 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:31.923136 11207 heartbeater.cc:511] Master 127.15.84.62:44703 requested a full tablet report, sending...
I20260810 14:44:31.924917 10920 ts_manager.cc:194] Registered new tserver with Master: 20149bfd33d44d1e96c59cef2c5a9b3c (127.15.84.1:45939)
I20260810 14:44:31.926496 10920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.84.1:46135
W20260810 14:44:32.044513 11211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:32.044837 11211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:32.044879 11211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:32.049036 11211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:32.049118 11211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:32.049213 11211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:44:32.054551 11211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:40285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:44703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:32.055703 11211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:32.057219 11211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:32.067782 11219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:32.067832 11217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:32.067829 11221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:32.068006 11211 server_base.cc:1034] running on GCE node
I20260810 14:44:32.068578 11211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:32.069286 11211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:32.070447 11211 hybrid_clock.cc:648] HybridClock initialized: now 1786373072070423 us; error 40 us; skew 500 ppm
I20260810 14:44:32.072455 11211 webserver.cc:460] Webserver started at http://127.15.84.2:46405/ using document root <none> and password file <none>
I20260810 14:44:32.073033 11211 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:32.073104 11211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:32.073354 11211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:32.075749 11211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "256eef552c4c4600827ccdeb0890df1b"
format_stamp: "Formatted at 2026-08-10 14:44:32 on dist-test-slave-rmhg"
server_key: "b60d16745662da1f9ecaadc1eddbbb5a"
server_key_iv: "68ab840c819267dc1fe5dbd94f5f8695"
server_key_version: "encryptionkey@0"
I20260810 14:44:32.076393 11211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "256eef552c4c4600827ccdeb0890df1b"
format_stamp: "Formatted at 2026-08-10 14:44:32 on dist-test-slave-rmhg"
server_key: "b60d16745662da1f9ecaadc1eddbbb5a"
server_key_iv: "68ab840c819267dc1fe5dbd94f5f8695"
server_key_version: "encryptionkey@0"
I20260810 14:44:32.080570 11211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:44:32.083563 11232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:32.085071 11211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:44:32.085283 11211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "256eef552c4c4600827ccdeb0890df1b"
format_stamp: "Formatted at 2026-08-10 14:44:32 on dist-test-slave-rmhg"
server_key: "b60d16745662da1f9ecaadc1eddbbb5a"
server_key_iv: "68ab840c819267dc1fe5dbd94f5f8695"
server_key_version: "encryptionkey@0"
I20260810 14:44:32.085414 11211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:32.095960 11211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:32.096705 11211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:32.096873 11211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:32.097553 11211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:32.098590 11211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:32.098642 11211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:32.098685 11211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:32.098706 11211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:32.107569 11211 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:44631
I20260810 14:44:32.107590 11409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:44631 every 8 connection(s)
I20260810 14:44:32.108567 11211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:44:32.110914 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 11211
I20260810 14:44:32.111035 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:44:32.111315 15696 external_mini_cluster.cc:1428] Setting key 9c273c5e7c48f035b4e087ebc7f19170
I20260810 14:44:32.113380 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:44703
--builtin_ntp_servers=127.15.84.20:40285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:44:32.119755 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.110640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:58231 (local address 127.15.84.62:44703)
0810 14:44:32.110818 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:32.110822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:32.112167 (+ 1345us) server_negotiation.cc:407] Connection header received
0810 14:44:32.113777 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:32.113780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:32.113838 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:32.113925 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:32.114954 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.115614 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.117285 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.117601 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.117719 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.119199 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:32.119203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:32.119207 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:32.119225 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:32.119253 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:32.119265 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:32.119345 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:32.119489 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 14:44:32.119581 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 14:44:32.120741 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.111149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44703 (local address 127.15.84.2:58231)
0810 14:44:32.111956 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:32.112046 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:32.113533 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:32.114119 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:32.114127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:32.114228 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:32.114799 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:32.114815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.115780 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.115784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:32.117105 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:32.117119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.117776 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.117779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:32.117956 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.118733 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:44:32.118735 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:32.118786 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:44:32.118998 (+ 212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:32.119356 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:32.119362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:32.119366 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:44:32.119467 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 14:44:32.119756 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":155,"threads_started":1}
I20260810 14:44:32.122120 11410 heartbeater.cc:347] Connected to a master server at 127.15.84.62:44703
I20260810 14:44:32.122428 11410 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:32.123188 11410 heartbeater.cc:511] Master 127.15.84.62:44703 requested a full tablet report, sending...
I20260810 14:44:32.124517 10920 ts_manager.cc:194] Registered new tserver with Master: 256eef552c4c4600827ccdeb0890df1b (127.15.84.2:44631)
I20260810 14:44:32.125142 10920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.84.2:58231
W20260810 14:44:32.237458 11414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:32.237797 11414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:32.237859 11414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:32.242133 11414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:32.242210 11414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:32.242295 11414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:44:32.247454 11414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:40285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:44703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:32.248732 11414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:32.250305 11414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:32.260663 11420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:32.260672 11421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:32.260726 11414 server_base.cc:1034] running on GCE node
W20260810 14:44:32.260646 11424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:32.261384 11414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:32.262090 11414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:32.263276 11414 hybrid_clock.cc:648] HybridClock initialized: now 1786373072263249 us; error 40 us; skew 500 ppm
I20260810 14:44:32.265358 11414 webserver.cc:460] Webserver started at http://127.15.84.3:33229/ using document root <none> and password file <none>
I20260810 14:44:32.265975 11414 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:32.266043 11414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:32.266258 11414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:32.268457 11414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "28e91f98e3a04335aaac3b199639cad7"
format_stamp: "Formatted at 2026-08-10 14:44:32 on dist-test-slave-rmhg"
server_key: "ea21e9302fe2cd1951e487e33e05b3bc"
server_key_iv: "6ae7c5d9377fc845e01ac32d4201c9af"
server_key_version: "encryptionkey@0"
I20260810 14:44:32.269055 11414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "28e91f98e3a04335aaac3b199639cad7"
format_stamp: "Formatted at 2026-08-10 14:44:32 on dist-test-slave-rmhg"
server_key: "ea21e9302fe2cd1951e487e33e05b3bc"
server_key_iv: "6ae7c5d9377fc845e01ac32d4201c9af"
server_key_version: "encryptionkey@0"
I20260810 14:44:32.273069 11414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:44:32.275995 11434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:32.277455 11414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 14:44:32.277580 11414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "28e91f98e3a04335aaac3b199639cad7"
format_stamp: "Formatted at 2026-08-10 14:44:32 on dist-test-slave-rmhg"
server_key: "ea21e9302fe2cd1951e487e33e05b3bc"
server_key_iv: "6ae7c5d9377fc845e01ac32d4201c9af"
server_key_version: "encryptionkey@0"
I20260810 14:44:32.277706 11414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:32.304738 11414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:32.305581 11414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:32.305788 11414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:32.306429 11414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:32.307437 11414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:32.307485 11414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:32.307528 11414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:32.307559 11414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:32.316359 11414 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:42643
I20260810 14:44:32.316437 11599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:42643 every 8 connection(s)
I20260810 14:44:32.317430 11414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:44:32.322365 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 11414
I20260810 14:44:32.322500 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:44:32.322806 15696 external_mini_cluster.cc:1428] Setting key c00bc31a05c8e7337bceadc9142f9996
I20260810 14:44:32.332670 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.319400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:46403 (local address 127.15.84.62:44703)
0810 14:44:32.319548 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:32.319552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:32.320777 (+ 1225us) server_negotiation.cc:407] Connection header received
0810 14:44:32.322598 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:32.322602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:32.322689 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:32.322752 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:32.324335 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.325509 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.330620 (+ 5111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.330875 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.330974 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.332232 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:32.332235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:32.332237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:32.332254 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:32.332278 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:32.332287 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:32.332353 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:32.332477 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 14:44:32.332542 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:44:32.333755 11604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.319864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:44703 (local address 127.15.84.3:46403)
0810 14:44:32.320607 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:32.320663 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:32.322264 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:32.323251 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:32.323262 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:32.323398 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:32.324163 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:32.324185 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.329183 (+ 4998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.329189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:32.330468 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:32.330486 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.331033 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.331036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:32.331208 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.331884 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:44:32.331886 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:32.331939 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:44:32.332025 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:32.332360 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:32.332365 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:32.332369 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:44:32.332470 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 14:44:32.332790 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
I20260810 14:44:32.335134 11600 heartbeater.cc:347] Connected to a master server at 127.15.84.62:44703
I20260810 14:44:32.335417 11600 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:32.336015 11600 heartbeater.cc:511] Master 127.15.84.62:44703 requested a full tablet report, sending...
I20260810 14:44:32.337198 10920 ts_manager.cc:194] Registered new tserver with Master: 28e91f98e3a04335aaac3b199639cad7 (127.15.84.3:42643)
I20260810 14:44:32.337800 10920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.84.3:46403
I20260810 14:44:32.338856 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:44:32.344317 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.340909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.15.84.62:44703)
0810 14:44:32.341045 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:32.341049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:32.341242 (+ 193us) server_negotiation.cc:407] Connection header received
0810 14:44:32.341357 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:32.341360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:32.341404 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:32.341467 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:32.341707 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.342228 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.342971 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.343192 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.343261 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.343937 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:32.343940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:32.343942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:32.343954 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:32.343973 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:32.343981 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:32.344031 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:32.344095 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 14:44:32.344154 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:44:32.347203 10920 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:44:32.350330 10920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:44:32.365119 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.359020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:42643 (local address 127.0.0.1:46874)
0810 14:44:32.360258 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:32.360271 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:32.360354 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:32.360889 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:32.360895 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:32.360917 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:32.361058 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:32.361065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.362149 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.362152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:32.363587 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:32.363597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.364479 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.364483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:32.364611 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.364695 (+ 84us) client_negotiation.cc:769] Sending connection context
0810 14:44:32.364821 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 14:44:32.364957 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"mutex_wait_us":18,"thread_start_us":131,"threads_started":1}
I20260810 14:44:32.365185 11627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.359760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.15.84.3:42643)
0810 14:44:32.360107 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:32.360120 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:32.360447 (+ 327us) server_negotiation.cc:407] Connection header received
0810 14:44:32.360533 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:32.360539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:32.360740 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:32.360882 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:32.361192 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.362044 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.363722 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.364337 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.364446 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.364685 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:32.364866 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 14:44:32.364979 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":115,"threads_started":1}
I20260810 14:44:32.365784 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.359230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:44631 (local address 127.0.0.1:42460)
0810 14:44:32.360473 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:32.360486 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:32.360576 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:32.361209 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:32.361212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:32.361230 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:32.361309 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:32.361314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.362715 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.362718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:32.364118 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:32.364128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.365451 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.365455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:32.365544 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.365564 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 14:44:32.365625 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 14:44:32.365683 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"mutex_wait_us":101,"thread_start_us":84,"threads_started":1}
I20260810 14:44:32.365978 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.359772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42460 (local address 127.15.84.2:44631)
0810 14:44:32.360328 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:32.360340 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:32.360668 (+ 328us) server_negotiation.cc:407] Connection header received
0810 14:44:32.360750 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:32.360754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:32.360922 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:32.361050 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:32.361429 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.362574 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.364258 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.365038 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.365142 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.365384 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:32.365707 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 14:44:32.365821 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":148,"threads_started":1}
I20260810 14:44:32.366781 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.358921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:45939 (local address 127.0.0.1:54942)
0810 14:44:32.359846 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:32.359886 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:32.360016 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:32.361053 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:32.361056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:32.361074 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:32.361220 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:32.361228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.362715 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.362718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:32.364116 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:32.364128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.366395 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.366399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:32.366496 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.366520 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:44:32.366584 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 14:44:32.366656 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":111,"thread_start_us":77,"threads_started":1}
I20260810 14:44:32.367399 11624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.359068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.15.84.1:45939)
0810 14:44:32.360357 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:32.360367 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:32.360394 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:44:32.360478 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:32.360484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:32.360732 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:32.360881 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:32.361341 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.362574 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.364286 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.365079 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.365214 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.367151 (+ 1937us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:32.367197 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 14:44:32.367275 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1195,"thread_start_us":130,"threads_started":1}
I20260810 14:44:32.368827 11309 tablet_service.cc:1467] Processing CreateTablet for tablet 468d19e3f78c4e53882ed70400696f99 (DEFAULT_TABLE table=test-table [id=6947acaf084e44c89dbe851de43c4cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:32.368815 11503 tablet_service.cc:1467] Processing CreateTablet for tablet 468d19e3f78c4e53882ed70400696f99 (DEFAULT_TABLE table=test-table [id=6947acaf084e44c89dbe851de43c4cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:32.370081 11309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468d19e3f78c4e53882ed70400696f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:32.370081 11503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468d19e3f78c4e53882ed70400696f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:32.369932 11103 tablet_service.cc:1467] Processing CreateTablet for tablet 468d19e3f78c4e53882ed70400696f99 (DEFAULT_TABLE table=test-table [id=6947acaf084e44c89dbe851de43c4cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:32.371016 11103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468d19e3f78c4e53882ed70400696f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:32.379940 11629 tablet_bootstrap.cc:492] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7: Bootstrap starting.
I20260810 14:44:32.380349 11630 tablet_bootstrap.cc:492] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c: Bootstrap starting.
I20260810 14:44:32.382797 11629 tablet_bootstrap.cc:654] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:32.383051 11630 tablet_bootstrap.cc:654] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:32.383827 11630 log.cc:824] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:32.383824 11629 log.cc:824] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:32.384630 11631 tablet_bootstrap.cc:492] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b: Bootstrap starting.
I20260810 14:44:32.386055 11630 tablet_bootstrap.cc:492] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c: No bootstrap required, opened a new log
I20260810 14:44:32.386292 11630 ts_tablet_manager.cc:1397] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:44:32.387718 11631 tablet_bootstrap.cc:654] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:32.388728 11631 log.cc:824] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:32.390693 11630 raft_consensus.cc:348] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 45939 } } peers { permanent_uuid: "28e91f98e3a04335aaac3b199639cad7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42643 } } peers { permanent_uuid: "256eef552c4c4600827ccdeb0890df1b" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44631 } }
I20260810 14:44:32.390955 11630 raft_consensus.cc:374] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:32.391012 11630 raft_consensus.cc:729] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20149bfd33d44d1e96c59cef2c5a9b3c, State: Initialized, Role: FOLLOWER
I20260810 14:44:32.391204 11629 tablet_bootstrap.cc:492] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7: No bootstrap required, opened a new log
I20260810 14:44:32.391203 11631 tablet_bootstrap.cc:492] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b: No bootstrap required, opened a new log
I20260810 14:44:32.391423 11631 ts_tablet_manager.cc:1397] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260810 14:44:32.391423 11629 ts_tablet_manager.cc:1397] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260810 14:44:32.391499 11630 consensus_queue.cc:260] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 45939 } } peers { permanent_uuid: "28e91f98e3a04335aaac3b199639cad7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42643 } } peers { permanent_uuid: "256eef552c4c4600827ccdeb0890df1b" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44631 } }
I20260810 14:44:32.392249 11207 heartbeater.cc:503] Master 127.15.84.62:44703 was elected leader, sending a full tablet report...
I20260810 14:44:32.393698 11630 ts_tablet_manager.cc:1428] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 14:44:32.395897 11629 raft_consensus.cc:348] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 45939 } } peers { permanent_uuid: "28e91f98e3a04335aaac3b199639cad7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42643 } } peers { permanent_uuid: "256eef552c4c4600827ccdeb0890df1b" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44631 } }
I20260810 14:44:32.395896 11631 raft_consensus.cc:348] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 45939 } } peers { permanent_uuid: "28e91f98e3a04335aaac3b199639cad7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42643 } } peers { permanent_uuid: "256eef552c4c4600827ccdeb0890df1b" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44631 } }
I20260810 14:44:32.396198 11629 raft_consensus.cc:374] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:32.396198 11631 raft_consensus.cc:374] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:32.396262 11629 raft_consensus.cc:729] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e91f98e3a04335aaac3b199639cad7, State: Initialized, Role: FOLLOWER
I20260810 14:44:32.396271 11631 raft_consensus.cc:729] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 256eef552c4c4600827ccdeb0890df1b, State: Initialized, Role: FOLLOWER
I20260810 14:44:32.396808 11631 consensus_queue.cc:260] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 45939 } } peers { permanent_uuid: "28e91f98e3a04335aaac3b199639cad7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42643 } } peers { permanent_uuid: "256eef552c4c4600827ccdeb0890df1b" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44631 } }
I20260810 14:44:32.396947 11629 consensus_queue.cc:260] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 45939 } } peers { permanent_uuid: "28e91f98e3a04335aaac3b199639cad7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42643 } } peers { permanent_uuid: "256eef552c4c4600827ccdeb0890df1b" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44631 } }
I20260810 14:44:32.397634 11600 heartbeater.cc:503] Master 127.15.84.62:44703 was elected leader, sending a full tablet report...
I20260810 14:44:32.397958 11631 ts_tablet_manager.cc:1428] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260810 14:44:32.397958 11629 ts_tablet_manager.cc:1428] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 14:44:32.398355 11410 heartbeater.cc:503] Master 127.15.84.62:44703 was elected leader, sending a full tablet report...
W20260810 14:44:32.410725 11208 tablet.cc:2369] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:44:32.430600 11635 raft_consensus.cc:482] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:44:32.430881 11635 raft_consensus.cc:504] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 45939 } } peers { permanent_uuid: "28e91f98e3a04335aaac3b199639cad7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42643 } } peers { permanent_uuid: "256eef552c4c4600827ccdeb0890df1b" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44631 } }
I20260810 14:44:32.432111 11635 leader_election.cc:288] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28e91f98e3a04335aaac3b199639cad7 (127.15.84.3:42643), 256eef552c4c4600827ccdeb0890df1b (127.15.84.2:44631)
I20260810 14:44:32.437789 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.432526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:55447 (local address 127.15.84.2:44631)
0810 14:44:32.432663 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:32.432667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:32.432922 (+ 255us) server_negotiation.cc:407] Connection header received
0810 14:44:32.433094 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:32.433097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:32.433203 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:32.433276 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:32.433760 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.434627 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.436576 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.437207 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.437303 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.437357 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:32.437560 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 14:44:32.437651 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 14:44:32.437789 11627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.432526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:52879 (local address 127.15.84.3:42643)
0810 14:44:32.432661 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:32.432665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:32.433049 (+ 384us) server_negotiation.cc:407] Connection header received
0810 14:44:32.433113 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:32.433115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:32.433189 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:32.433273 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:32.433760 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.434628 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.436453 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.437079 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.437254 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.437329 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:32.437560 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 14:44:32.437644 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:44:32.437870 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.432412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:42643 (local address 127.15.84.1:52879)
0810 14:44:32.432943 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:32.432955 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:32.433042 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:32.433297 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:32.433300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:32.433344 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:32.433594 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:32.433604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.434760 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.434764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:32.436290 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:32.436305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.437257 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.437261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:32.437380 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.437462 (+ 82us) client_negotiation.cc:769] Sending connection context
0810 14:44:32.437544 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 14:44:32.437648 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":223,"spinlock_wait_cycles":43392,"thread_start_us":99,"threads_started":1}
I20260810 14:44:32.438254 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.432412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:44631 (local address 127.15.84.1:55447)
0810 14:44:32.432820 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:32.432835 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:32.432974 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:32.433297 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:32.433300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:32.433346 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:32.433598 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:32.433604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.434760 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.434764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:32.436290 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:32.436305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.437301 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.437304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:32.437385 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.437459 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 14:44:32.437537 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:44:32.437648 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":166,"spinlock_wait_cycles":4736,"thread_start_us":122,"threads_started":1}
I20260810 14:44:32.438572 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468d19e3f78c4e53882ed70400696f99" candidate_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28e91f98e3a04335aaac3b199639cad7" is_pre_election: true
I20260810 14:44:32.438934 11531 raft_consensus.cc:2454] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20149bfd33d44d1e96c59cef2c5a9b3c in term 0.
I20260810 14:44:32.438956 11340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468d19e3f78c4e53882ed70400696f99" candidate_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "256eef552c4c4600827ccdeb0890df1b" is_pre_election: true
I20260810 14:44:32.439288 11340 raft_consensus.cc:2454] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20149bfd33d44d1e96c59cef2c5a9b3c in term 0.
I20260810 14:44:32.439512 11040 leader_election.cc:302] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20149bfd33d44d1e96c59cef2c5a9b3c, 28e91f98e3a04335aaac3b199639cad7; no voters:
I20260810 14:44:32.439829 11635 raft_consensus.cc:2781] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:44:32.439935 11635 raft_consensus.cc:482] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:44:32.439989 11635 raft_consensus.cc:3037] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:32.441180 11635 raft_consensus.cc:504] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 45939 } } peers { permanent_uuid: "28e91f98e3a04335aaac3b199639cad7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42643 } } peers { permanent_uuid: "256eef552c4c4600827ccdeb0890df1b" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44631 } }
I20260810 14:44:32.441555 11635 leader_election.cc:288] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [CANDIDATE]: Term 1 election: Requested vote from peers 28e91f98e3a04335aaac3b199639cad7 (127.15.84.3:42643), 256eef552c4c4600827ccdeb0890df1b (127.15.84.2:44631)
I20260810 14:44:32.441937 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468d19e3f78c4e53882ed70400696f99" candidate_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28e91f98e3a04335aaac3b199639cad7"
I20260810 14:44:32.441958 11340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468d19e3f78c4e53882ed70400696f99" candidate_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "256eef552c4c4600827ccdeb0890df1b"
I20260810 14:44:32.442060 11531 raft_consensus.cc:3037] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:32.442075 11340 raft_consensus.cc:3037] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:32.443106 11531 raft_consensus.cc:2454] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20149bfd33d44d1e96c59cef2c5a9b3c in term 1.
I20260810 14:44:32.443111 11340 raft_consensus.cc:2454] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20149bfd33d44d1e96c59cef2c5a9b3c in term 1.
I20260810 14:44:32.443413 11040 leader_election.cc:302] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20149bfd33d44d1e96c59cef2c5a9b3c, 28e91f98e3a04335aaac3b199639cad7; no voters:
I20260810 14:44:32.443588 11635 raft_consensus.cc:2781] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:32.443817 11635 raft_consensus.cc:686] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [term 1 LEADER]: Becoming Leader. State: Replica: 20149bfd33d44d1e96c59cef2c5a9b3c, State: Running, Role: LEADER
I20260810 14:44:32.444144 11635 consensus_queue.cc:237] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 45939 } } peers { permanent_uuid: "28e91f98e3a04335aaac3b199639cad7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42643 } } peers { permanent_uuid: "256eef552c4c4600827ccdeb0890df1b" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44631 } }
I20260810 14:44:32.447639 10919 catalog_manager.cc:5495] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 20149bfd33d44d1e96c59cef2c5a9b3c (127.15.84.1). New cstate: current_term: 1 leader_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20149bfd33d44d1e96c59cef2c5a9b3c" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 45939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28e91f98e3a04335aaac3b199639cad7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "256eef552c4c4600827ccdeb0890df1b" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44631 } health_report { overall_health: UNKNOWN } } }
I20260810 14:44:32.465055 11624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.461388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.15.84.1:45939)
0810 14:44:32.461531 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:32.461535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:32.461549 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:44:32.461674 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:32.461676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:32.461740 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:32.461836 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:44:32.462119 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.462640 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.463556 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:32.463766 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:32.463833 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:32.464174 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:44:32.464654 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:44:32.464749 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:32.464846 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 14:44:32.464904 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:44:32.471359 11340 raft_consensus.cc:1261] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b [term 1 FOLLOWER]: Refusing update from remote peer 20149bfd33d44d1e96c59cef2c5a9b3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:44:32.471359 11531 raft_consensus.cc:1261] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7 [term 1 FOLLOWER]: Refusing update from remote peer 20149bfd33d44d1e96c59cef2c5a9b3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:44:32.472103 11635 consensus_queue.cc:1035] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28e91f98e3a04335aaac3b199639cad7" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 42643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:44:32.472318 11644 consensus_queue.cc:1035] T 468d19e3f78c4e53882ed70400696f99 P 20149bfd33d44d1e96c59cef2c5a9b3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "256eef552c4c4600827ccdeb0890df1b" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 44631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:44:32.490056 10920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:44:32.490293 10920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:44:32.493036 10920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5adbe28f834f44f387e8258f534f562a: Sending DeleteTablet for 3 replicas of tablet 468d19e3f78c4e53882ed70400696f99
I20260810 14:44:32.493937 11503 tablet_service.cc:1514] Processing DeleteTablet for tablet 468d19e3f78c4e53882ed70400696f99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:32 UTC) from {username='slave'} at 127.0.0.1:46874
I20260810 14:44:32.493916 11103 tablet_service.cc:1514] Processing DeleteTablet for tablet 468d19e3f78c4e53882ed70400696f99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:32 UTC) from {username='slave'} at 127.0.0.1:54942
I20260810 14:44:32.494670 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 11015
I20260810 14:44:32.494709 11309 tablet_service.cc:1514] Processing DeleteTablet for tablet 468d19e3f78c4e53882ed70400696f99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:32 UTC) from {username='slave'} at 127.0.0.1:42460
I20260810 14:44:32.498756 11669 tablet_replica.cc:330] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7: stopping tablet replica
I20260810 14:44:32.499104 11669 raft_consensus.cc:2229] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:44:32.499447 11669 raft_consensus.cc:2258] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:44:32.501421 11670 tablet_replica.cc:330] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b: stopping tablet replica
I20260810 14:44:32.501561 11669 ts_tablet_manager.cc:1905] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:44:32.501803 11670 raft_consensus.cc:2229] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:44:32.502183 11670 raft_consensus.cc:2258] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:44:32.503954 11670 ts_tablet_manager.cc:1905] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:44:32.504659 11669 ts_tablet_manager.cc:1918] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:44:32.504771 11669 log.cc:1196] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/468d19e3f78c4e53882ed70400696f99
I20260810 14:44:32.505107 11669 ts_tablet_manager.cc:1939] T 468d19e3f78c4e53882ed70400696f99 P 28e91f98e3a04335aaac3b199639cad7: Deleting consensus metadata
I20260810 14:44:32.506707 10897 catalog_manager.cc:4841] TS 28e91f98e3a04335aaac3b199639cad7 (127.15.84.3:42643): tablet 468d19e3f78c4e53882ed70400696f99 (table test-table [id=6947acaf084e44c89dbe851de43c4cee]) successfully deleted
I20260810 14:44:32.506757 11670 ts_tablet_manager.cc:1918] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:44:32.506855 11670 log.cc:1196] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/468d19e3f78c4e53882ed70400696f99
I20260810 14:44:32.507134 11670 ts_tablet_manager.cc:1939] T 468d19e3f78c4e53882ed70400696f99 P 256eef552c4c4600827ccdeb0890df1b: Deleting consensus metadata
W20260810 14:44:32.507587 10898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:45939 (error 108)
I20260810 14:44:32.507722 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 11211
I20260810 14:44:32.513036 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.512572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:45939 (local address 127.0.0.1:54964)
0810 14:44:32.512866 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:32.512943 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:45939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
W20260810 14:44:32.513329 10898 catalog_manager.cc:4584] TS 20149bfd33d44d1e96c59cef2c5a9b3c (127.15.84.1:45939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 468d19e3f78c4e53882ed70400696f99: Network error: Client connection negotiation failed: client connection to 127.15.84.1:45939: connect: Connection refused (error 111)
I20260810 14:44:32.520457 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 11414
I20260810 14:44:32.523363 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:32.520764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:44631 (local address 127.0.0.1:42468)
0810 14:44:32.523168 (+ 2404us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:32.523248 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.2:44631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2276}
I20260810 14:44:32.534091 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 10873
2026-08-10T14:44:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1104 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:32 dist-test-slave-rmhg krb5kdc[11678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:32 dist-test-slave-rmhg krb5kdc[11678](info): set up 2 sockets
Aug 10 14:44:32 dist-test-slave-rmhg krb5kdc[11678](info): commencing operation
krb5kdc: starting...
W20260810 14:44:33.706535 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:33 dist-test-slave-rmhg krb5kdc[11678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:44:33.838943 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40453
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:40453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:33.956948 11697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:33.957288 11697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:33.957350 11697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:33.961412 11697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:33.961488 11697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:33.961512 11697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:33.961534 11697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:33.961553 11697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:33.967082 11697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:40453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:33.968257 11697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:33.969386 11697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:33.978305 11708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:33.978286 11707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:33.978411 11697 server_base.cc:1034] running on GCE node
W20260810 14:44:33.978286 11712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:33.979210 11697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:33.980429 11697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:33.981590 11697 hybrid_clock.cc:648] HybridClock initialized: now 1786373073981574 us; error 35 us; skew 500 ppm
Aug 10 14:44:33 dist-test-slave-rmhg krb5kdc[11678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373073, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:33.990444 11697 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:44:33.991500 11697 webserver.cc:460] Webserver started at http://127.15.84.62:42277/ using document root <none> and password file <none>
I20260810 14:44:33.992071 11697 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:33.992133 11697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:33.992328 11697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:33.994560 11697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "95b5660219e8455abbfbddf08f03ef35"
format_stamp: "Formatted at 2026-08-10 14:44:33 on dist-test-slave-rmhg"
server_key: "28db206554a12dac31237346aaddf62a"
server_key_iv: "1b98b29878f849ed9f1cb2edc08fcd16"
server_key_version: "encryptionkey@0"
I20260810 14:44:33.995105 11697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "95b5660219e8455abbfbddf08f03ef35"
format_stamp: "Formatted at 2026-08-10 14:44:33 on dist-test-slave-rmhg"
server_key: "28db206554a12dac31237346aaddf62a"
server_key_iv: "1b98b29878f849ed9f1cb2edc08fcd16"
server_key_version: "encryptionkey@0"
I20260810 14:44:33.999027 11697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:44:34.001598 11720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:34.002933 11697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:44:34.003047 11697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "95b5660219e8455abbfbddf08f03ef35"
format_stamp: "Formatted at 2026-08-10 14:44:33 on dist-test-slave-rmhg"
server_key: "28db206554a12dac31237346aaddf62a"
server_key_iv: "1b98b29878f849ed9f1cb2edc08fcd16"
server_key_version: "encryptionkey@0"
I20260810 14:44:34.003156 11697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:34.028177 11697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:34.031302 11697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:34.031479 11697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:34.038842 11697 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:40453
I20260810 14:44:34.038873 11791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:40453 every 8 connection(s)
I20260810 14:44:34.039886 11697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:34.042943 11792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:34.047343 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 11697
I20260810 14:44:34.047518 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:34.047837 15696 external_mini_cluster.cc:1428] Setting key 02f10a4f7e8b07861b09596c80f7dc00
I20260810 14:44:34.049592 11792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35: Bootstrap starting.
I20260810 14:44:34.051968 11792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:34.052731 11792 log.cc:824] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:34.055032 11792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35: No bootstrap required, opened a new log
Aug 10 14:44:34 dist-test-slave-rmhg krb5kdc[11678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:44:34.059154 11792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b5660219e8455abbfbddf08f03ef35" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40453 } }
I20260810 14:44:34.059403 11792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:34.059442 11792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b5660219e8455abbfbddf08f03ef35, State: Initialized, Role: FOLLOWER
I20260810 14:44:34.059929 11792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b5660219e8455abbfbddf08f03ef35" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40453 } }
I20260810 14:44:34.060061 11792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:34.060123 11792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:34.060225 11792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:34.061440 11792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b5660219e8455abbfbddf08f03ef35" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40453 } }
I20260810 14:44:34.061815 11792 leader_election.cc:302] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95b5660219e8455abbfbddf08f03ef35; no voters:
I20260810 14:44:34.062130 11792 leader_election.cc:288] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:34.062258 11802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:34.062477 11802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [term 1 LEADER]: Becoming Leader. State: Replica: 95b5660219e8455abbfbddf08f03ef35, State: Running, Role: LEADER
I20260810 14:44:34.062824 11802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b5660219e8455abbfbddf08f03ef35" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40453 } }
I20260810 14:44:34.063421 11792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:34.063676 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:34.049466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.15.84.62:40453)
0810 14:44:34.049935 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:34.049952 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:34.049991 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:44:34.050663 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:34.050681 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:34.051040 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:34.051370 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:34.052221 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.053399 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.054568 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.055077 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.055242 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:34.058004 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:34.058010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:34.058024 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:34.058057 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:34.059769 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:34.060265 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:34.060271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:34.060277 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:34.060430 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:34.060693 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:34.060696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:34.060697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:34.061309 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:34.061510 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:34.062099 (+ 589us) server_negotiation.cc:299] Negotiation successful
0810 14:44:34.062461 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":146,"threads_started":1}
I20260810 14:44:34.064611 11805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95b5660219e8455abbfbddf08f03ef35. Latest consensus state: current_term: 1 leader_uuid: "95b5660219e8455abbfbddf08f03ef35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b5660219e8455abbfbddf08f03ef35" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40453 } } }
I20260810 14:44:34.064826 11805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:34.065243 11804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95b5660219e8455abbfbddf08f03ef35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b5660219e8455abbfbddf08f03ef35" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40453 } } }
I20260810 14:44:34.065397 11804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35 [sys.catalog]: This master's current role is: LEADER
W20260810 14:44:34.068612 11825 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:44:34.068755 11825 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:44:34.068905 11816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:34.069893 11816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:34.075901 11816 catalog_manager.cc:1347] Generated new cluster ID: 837e74a913924d59aedec2770c39ea7b
I20260810 14:44:34.075991 11816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:34.086503 11816 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:34.087539 11816 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:34.095006 11816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35: Generated new TSK 0
I20260810 14:44:34.095897 11816 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:44:34.133301 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40453
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:34.252641 11834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:34.252936 11834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:34.252981 11834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:34.257073 11834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:34.257186 11834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:34.257258 11834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:44:34.262744 11834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:34.263942 11834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:34.265075 11834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:34.275352 11841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:34.275471 11834 server_base.cc:1034] running on GCE node
W20260810 14:44:34.275345 11843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:34.275345 11846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:34.276130 11834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:34.276858 11834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:34.278031 11834 hybrid_clock.cc:648] HybridClock initialized: now 1786373074277985 us; error 62 us; skew 500 ppm
Aug 10 14:44:34 dist-test-slave-rmhg krb5kdc[11678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373074, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:34.281006 11834 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:44:34.282039 11834 webserver.cc:460] Webserver started at http://127.15.84.1:35947/ using document root <none> and password file <none>
I20260810 14:44:34.282629 11834 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:34.282697 11834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:34.282938 11834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:34.285393 11834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "8173e8c9abb347a4b6c75e9b76d5e71e"
format_stamp: "Formatted at 2026-08-10 14:44:34 on dist-test-slave-rmhg"
server_key: "0fb67dc7f10a17135c6f8c277f49a238"
server_key_iv: "3d83fc4cfdbed89be90f45cc1975935c"
server_key_version: "encryptionkey@0"
I20260810 14:44:34.285938 11834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "8173e8c9abb347a4b6c75e9b76d5e71e"
format_stamp: "Formatted at 2026-08-10 14:44:34 on dist-test-slave-rmhg"
server_key: "0fb67dc7f10a17135c6f8c277f49a238"
server_key_iv: "3d83fc4cfdbed89be90f45cc1975935c"
server_key_version: "encryptionkey@0"
I20260810 14:44:34.289772 11834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:44:34.292270 11861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:34.293608 11834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:44:34.293718 11834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "8173e8c9abb347a4b6c75e9b76d5e71e"
format_stamp: "Formatted at 2026-08-10 14:44:34 on dist-test-slave-rmhg"
server_key: "0fb67dc7f10a17135c6f8c277f49a238"
server_key_iv: "3d83fc4cfdbed89be90f45cc1975935c"
server_key_version: "encryptionkey@0"
I20260810 14:44:34.293829 11834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:34.315923 11834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:34.319007 11834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:34.319221 11834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:34.319881 11834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:34.320789 11834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:34.320839 11834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:34.320886 11834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:34.320915 11834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:34.330269 11834 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:42981
I20260810 14:44:34.330384 12035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:42981 every 8 connection(s)
I20260810 14:44:34.331295 11834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:44:34.331547 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 11834
I20260810 14:44:34.331699 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:44:34.332059 15696 external_mini_cluster.cc:1428] Setting key 259c57eddb203d397645a60d55638812
Aug 10 14:44:34 dist-test-slave-rmhg krb5kdc[11678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373074, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
I20260810 14:44:34.348608 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:34.333394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:49335 (local address 127.15.84.62:40453)
0810 14:44:34.333529 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:34.333534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:34.335264 (+ 1730us) server_negotiation.cc:407] Connection header received
0810 14:44:34.336584 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:34.336588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:34.336660 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:34.336749 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:34.338356 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.339162 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.341138 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.341455 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.341597 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:34.344827 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:34.344830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:34.344832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:34.344862 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:34.346180 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:34.346737 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:34.346739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:34.346740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:34.346856 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:34.347253 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:34.347257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:34.347259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:34.347491 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:34.347575 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:34.348294 (+ 719us) server_negotiation.cc:299] Negotiation successful
0810 14:44:34.348427 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:44:34.349795 12040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:34.333940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40453 (local address 127.15.84.1:49335)
0810 14:44:34.335019 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:34.335111 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:34.336246 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:34.337036 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:34.337046 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:34.337630 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:34.338194 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:34.338210 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.339338 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.339368 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:34.340967 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:34.340982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.341690 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.341694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:34.341939 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:34.342642 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:44:34.342668 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:34.344595 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:34.346312 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:34.346319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:34.346339 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:34.346632 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:34.346945 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:34.346947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:34.346949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:34.347146 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:34.347599 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:34.347607 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:34.347984 (+ 377us) client_negotiation.cc:769] Sending connection context
0810 14:44:34.348232 (+ 248us) client_negotiation.cc:240] Negotiation successful
0810 14:44:34.348615 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":150,"threads_started":1}
I20260810 14:44:34.351166 12036 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40453
I20260810 14:44:34.351420 12036 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:34.351955 12036 heartbeater.cc:511] Master 127.15.84.62:40453 requested a full tablet report, sending...
I20260810 14:44:34.353648 11739 ts_manager.cc:194] Registered new tserver with Master: 8173e8c9abb347a4b6c75e9b76d5e71e (127.15.84.1:42981)
I20260810 14:44:34.354832 11739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:49335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:44:34.364547 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40453
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:34.486166 12046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:34.486462 12046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:34.486509 12046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:34.490736 12046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:34.490819 12046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:34.490880 12046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:44:34.496780 12046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:34.498039 12046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:34.499169 12046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:34.509920 12056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:34.509931 12055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:34.510072 12046 server_base.cc:1034] running on GCE node
W20260810 14:44:34.510301 12058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:34.510761 12046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:34.511456 12046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:34.512634 12046 hybrid_clock.cc:648] HybridClock initialized: now 1786373074512612 us; error 39 us; skew 500 ppm
Aug 10 14:44:34 dist-test-slave-rmhg krb5kdc[11678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373074, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:34.515730 12046 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:44:34.516762 12046 webserver.cc:460] Webserver started at http://127.15.84.2:40625/ using document root <none> and password file <none>
I20260810 14:44:34.517426 12046 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:34.517504 12046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:34.517721 12046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:34.520088 12046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "c31ecc2825f0413191dcf9667e7ac9f0"
format_stamp: "Formatted at 2026-08-10 14:44:34 on dist-test-slave-rmhg"
server_key: "618523758e47b9213c4fe4147c7fd017"
server_key_iv: "ec8cf415847e5f906f6c7ba64c26c8f9"
server_key_version: "encryptionkey@0"
I20260810 14:44:34.520749 12046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "c31ecc2825f0413191dcf9667e7ac9f0"
format_stamp: "Formatted at 2026-08-10 14:44:34 on dist-test-slave-rmhg"
server_key: "618523758e47b9213c4fe4147c7fd017"
server_key_iv: "ec8cf415847e5f906f6c7ba64c26c8f9"
server_key_version: "encryptionkey@0"
I20260810 14:44:34.524940 12046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 14:44:34.527617 12069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:34.528968 12046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:44:34.529114 12046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "c31ecc2825f0413191dcf9667e7ac9f0"
format_stamp: "Formatted at 2026-08-10 14:44:34 on dist-test-slave-rmhg"
server_key: "618523758e47b9213c4fe4147c7fd017"
server_key_iv: "ec8cf415847e5f906f6c7ba64c26c8f9"
server_key_version: "encryptionkey@0"
I20260810 14:44:34.529277 12046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:34.540745 12046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:34.544196 12046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:34.544420 12046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:34.545095 12046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:34.546020 12046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:34.546070 12046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:34.546128 12046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:34.546156 12046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:34.555398 12046 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:43441
I20260810 14:44:34.555440 12229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:43441 every 8 connection(s)
I20260810 14:44:34.556367 12046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:44:34.562780 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 12046
I20260810 14:44:34.562901 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:44:34.563166 15696 external_mini_cluster.cc:1428] Setting key 4baf095fa46d930b1665ce3e5655fa3d
Aug 10 14:44:34 dist-test-slave-rmhg krb5kdc[11678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373074, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:44:34.570963 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:34.558341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:39441 (local address 127.15.84.62:40453)
0810 14:44:34.558487 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:34.558490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:34.559599 (+ 1109us) server_negotiation.cc:407] Connection header received
0810 14:44:34.560549 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:34.560553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:34.560612 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:34.560689 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:34.562014 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.562613 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.564200 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.564527 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.564672 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:34.567310 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:34.567313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:34.567315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:34.567338 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:34.568731 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:34.569327 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:34.569331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:34.569332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:34.569415 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:34.569810 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:34.569813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:34.569814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:34.570008 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:34.570095 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:34.570658 (+ 563us) server_negotiation.cc:299] Negotiation successful
0810 14:44:34.570796 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:44:34.571843 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:34.558754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40453 (local address 127.15.84.2:39441)
0810 14:44:34.559444 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:34.559488 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:34.560336 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:34.560876 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:34.560883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:34.561313 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:34.561865 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:34.561881 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.562742 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.562745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:34.564082 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:34.564094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.564741 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.564744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:34.564919 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:34.565528 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:44:34.565546 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:34.567142 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:34.568851 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:34.568857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:34.568875 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:34.569228 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:34.569504 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:34.569508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:34.569510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:34.569708 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:34.570114 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:34.570121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:34.570396 (+ 275us) client_negotiation.cc:769] Sending connection context
0810 14:44:34.570573 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 14:44:34.570934 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":136,"threads_started":1}
I20260810 14:44:34.573195 12231 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40453
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
I20260810 14:44:34.573472 12231 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:34.574046 12231 heartbeater.cc:511] Master 127.15.84.62:40453 requested a full tablet report, sending...
I20260810 14:44:34.575203 11739 ts_manager.cc:194] Registered new tserver with Master: c31ecc2825f0413191dcf9667e7ac9f0 (127.15.84.2:43441)
I20260810 14:44:34.575881 11739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:39441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:44:34.592660 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40453
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:37021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:34.709570 12240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:34.709854 12240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:34.709920 12240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:34.714082 12240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:34.714164 12240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:34.714248 12240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:44:34.719828 12240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:40453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:34.721086 12240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:34.722234 12240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:34.732349 12253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:34.732344 12251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:34.732338 12250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:34.732743 12240 server_base.cc:1034] running on GCE node
I20260810 14:44:34.733238 12240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:34.733937 12240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:34.735117 12240 hybrid_clock.cc:648] HybridClock initialized: now 1786373074735095 us; error 36 us; skew 500 ppm
Aug 10 14:44:34 dist-test-slave-rmhg krb5kdc[11678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373074, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:34.737948 12240 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:44:34.738942 12240 webserver.cc:460] Webserver started at http://127.15.84.3:46539/ using document root <none> and password file <none>
I20260810 14:44:34.739534 12240 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:34.739602 12240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:34.739817 12240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:34.742040 12240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "194fc540ade146c38a8e4893104f1f5d"
format_stamp: "Formatted at 2026-08-10 14:44:34 on dist-test-slave-rmhg"
server_key: "75997bfb2a09bf2f80e14db45fdbf8d8"
server_key_iv: "440ba7f9b19f010ae1ae37b6d28d602c"
server_key_version: "encryptionkey@0"
I20260810 14:44:34.742640 12240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "194fc540ade146c38a8e4893104f1f5d"
format_stamp: "Formatted at 2026-08-10 14:44:34 on dist-test-slave-rmhg"
server_key: "75997bfb2a09bf2f80e14db45fdbf8d8"
server_key_iv: "440ba7f9b19f010ae1ae37b6d28d602c"
server_key_version: "encryptionkey@0"
I20260810 14:44:34.746690 12240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:44:34.749431 12264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:34.750793 12240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:44:34.750922 12240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "194fc540ade146c38a8e4893104f1f5d"
format_stamp: "Formatted at 2026-08-10 14:44:34 on dist-test-slave-rmhg"
server_key: "75997bfb2a09bf2f80e14db45fdbf8d8"
server_key_iv: "440ba7f9b19f010ae1ae37b6d28d602c"
server_key_version: "encryptionkey@0"
I20260810 14:44:34.751087 12240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:34.764761 12240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:34.768110 12240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:34.768319 12240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:34.768981 12240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:34.769969 12240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:34.770030 12240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:34.770074 12240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:34.770102 12240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:34.779361 12240 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:39641
I20260810 14:44:34.779386 12427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:39641 every 8 connection(s)
I20260810 14:44:34.780417 12240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
Aug 10 14:44:34 dist-test-slave-rmhg krb5kdc[11678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373074, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:44:34.790760 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 12240
I20260810 14:44:34.790916 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:44:34.791227 15696 external_mini_cluster.cc:1428] Setting key 5fb351d100239505aacb679e75f1d2f2
I20260810 14:44:34.794787 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:34.782424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:58881 (local address 127.15.84.62:40453)
0810 14:44:34.782568 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:34.782572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:34.783738 (+ 1166us) server_negotiation.cc:407] Connection header received
0810 14:44:34.784768 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:34.784771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:34.784823 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:34.784887 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:34.786201 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.786766 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.787954 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.788188 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.788275 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:34.790919 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:34.790922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:34.790924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:34.790953 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:34.792540 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:34.793201 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:34.793204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:34.793205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:34.793293 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:34.793713 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:34.793716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:34.793717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:34.793930 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:34.794006 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:34.794530 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 14:44:34.794658 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:44:34.795680 12433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:34.782878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:40453 (local address 127.15.84.3:58881)
0810 14:44:34.783574 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:34.783620 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:34.784544 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:34.785060 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:34.785067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:34.785520 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:34.786053 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:34.786069 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.786876 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.786878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:34.787831 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:34.787841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.788321 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.788323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:34.788475 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:34.789087 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:44:34.789106 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:34.790719 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:34.792667 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:34.792674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:34.792696 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:34.793072 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:34.793380 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:34.793382 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:34.793384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:34.793600 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:34.794023 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:34.794029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:34.794323 (+ 294us) client_negotiation.cc:769] Sending connection context
0810 14:44:34.794498 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 14:44:34.794814 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":113,"threads_started":1}
I20260810 14:44:34.796823 12429 heartbeater.cc:347] Connected to a master server at 127.15.84.62:40453
I20260810 14:44:34.797060 12429 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:34.797680 12429 heartbeater.cc:511] Master 127.15.84.62:40453 requested a full tablet report, sending...
I20260810 14:44:34.798838 11739 ts_manager.cc:194] Registered new tserver with Master: 194fc540ade146c38a8e4893104f1f5d (127.15.84.3:39641)
I20260810 14:44:34.799445 11739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:58881
I20260810 14:44:34.805838 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:44:34.815378 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:34.808012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.15.84.62:40453)
0810 14:44:34.808186 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:34.808191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:34.808341 (+ 150us) server_negotiation.cc:407] Connection header received
0810 14:44:34.808450 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:34.808453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:34.808499 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:34.808568 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:34.809314 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.809885 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.810676 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.810889 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.810956 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:34.812513 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:34.812516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:34.812518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:34.812538 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:34.813769 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:34.814257 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:34.814259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:34.814261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:34.814337 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:34.814567 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:34.814569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:34.814570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:34.814784 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:34.814897 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:34.815115 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 14:44:34.815211 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:44:34.818724 11739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:44:34.822047 11739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:44:34.837697 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:34.831127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:42981 (local address 127.0.0.1:43962)
0810 14:44:34.831542 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:34.831604 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:34.831751 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:34.833016 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:34.833023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:34.833049 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:34.833265 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:34.833272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.834567 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.834571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:34.835847 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:34.835858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.837022 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.837026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:34.837180 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:34.837236 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 14:44:34.837368 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 14:44:34.837519 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":3,"thread_start_us":124,"threads_started":1}
I20260810 14:44:34.837692 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:34.831246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.15.84.1:42981)
0810 14:44:34.832407 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:34.832426 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:34.832454 (+ 28us) server_negotiation.cc:407] Connection header received
0810 14:44:34.832543 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:34.832550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:34.832833 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:34.833013 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:34.833395 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.834449 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.835975 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.836881 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.837003 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:34.837226 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:34.837412 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 14:44:34.837527 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":112,"threads_started":1}
I20260810 14:44:34.838130 12447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:34.831924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:39641 (local address 127.0.0.1:39926)
0810 14:44:34.832801 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:34.832814 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:34.832885 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:34.833819 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:34.833823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:34.833889 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:34.834019 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:34.834026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.835523 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.835527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:34.836496 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:34.836506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.837728 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.837732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:34.837831 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:34.837856 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:44:34.837925 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 14:44:34.837997 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":123,"spinlock_wait_cycles":896,"thread_start_us":115,"threads_started":1}
I20260810 14:44:34.838377 12445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:34.831206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:43441 (local address 127.0.0.1:57668)
0810 14:44:34.832612 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:34.832628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:34.832803 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:34.833816 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:34.833819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:34.833854 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:34.833982 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:34.833989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.835759 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.835762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:34.836897 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:34.836906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.838006 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.838010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:34.838106 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:34.838125 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 14:44:34.838168 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 14:44:34.838234 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1344,"spinlock_wait_cycles":5376,"thread_start_us":98,"threads_started":1}
I20260810 14:44:34.838610 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:34.832220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.15.84.3:39641)
0810 14:44:34.833108 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:34.833126 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:34.833189 (+ 63us) server_negotiation.cc:407] Connection header received
0810 14:44:34.833277 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:34.833283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:34.833486 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:34.833634 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:34.834156 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.835338 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.836626 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.837420 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.837516 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:34.837662 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:34.838324 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 14:44:34.838436 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":177,"threads_started":1}
I20260810 14:44:34.839107 12448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:34.832072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.15.84.2:43441)
0810 14:44:34.833376 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:34.833387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:34.833409 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:44:34.833481 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:34.833485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:34.833657 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:34.833872 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:34.834086 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.835338 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.837030 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:34.837865 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:34.838657 (+ 792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:34.838844 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:34.838894 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 14:44:34.838991 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1198,"thread_start_us":176,"threads_started":1}
I20260810 14:44:34.841703 12137 tablet_service.cc:1467] Processing CreateTablet for tablet 8f095b5adb2941d0affdf0ada9248e10 (DEFAULT_TABLE table=test-table [id=eef9d48a572848668637103943fbee57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:34.841689 11938 tablet_service.cc:1467] Processing CreateTablet for tablet 8f095b5adb2941d0affdf0ada9248e10 (DEFAULT_TABLE table=test-table [id=eef9d48a572848668637103943fbee57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:34.841949 12325 tablet_service.cc:1467] Processing CreateTablet for tablet 8f095b5adb2941d0affdf0ada9248e10 (DEFAULT_TABLE table=test-table [id=eef9d48a572848668637103943fbee57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:34.842828 12137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f095b5adb2941d0affdf0ada9248e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:34.842861 12325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f095b5adb2941d0affdf0ada9248e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:34.842856 11938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f095b5adb2941d0affdf0ada9248e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:34.850853 12453 tablet_bootstrap.cc:492] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0: Bootstrap starting.
I20260810 14:44:34.850915 12455 tablet_bootstrap.cc:492] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e: Bootstrap starting.
I20260810 14:44:34.852933 12455 tablet_bootstrap.cc:654] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:34.853341 12453 tablet_bootstrap.cc:654] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:34.853775 12455 log.cc:824] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:34.854125 12453 log.cc:824] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:34.854583 12454 tablet_bootstrap.cc:492] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d: Bootstrap starting.
I20260810 14:44:34.856494 12453 tablet_bootstrap.cc:492] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0: No bootstrap required, opened a new log
I20260810 14:44:34.856603 12455 tablet_bootstrap.cc:492] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e: No bootstrap required, opened a new log
I20260810 14:44:34.856731 12453 ts_tablet_manager.cc:1397] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:44:34.856766 12455 ts_tablet_manager.cc:1397] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260810 14:44:34.856812 12454 tablet_bootstrap.cc:654] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:34.857959 12454 log.cc:824] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:34.859803 12454 tablet_bootstrap.cc:492] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d: No bootstrap required, opened a new log
I20260810 14:44:34.860009 12454 ts_tablet_manager.cc:1397] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 14:44:34.860858 12453 raft_consensus.cc:348] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8173e8c9abb347a4b6c75e9b76d5e71e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42981 } } peers { permanent_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 43441 } } peers { permanent_uuid: "194fc540ade146c38a8e4893104f1f5d" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39641 } }
I20260810 14:44:34.861222 12453 raft_consensus.cc:374] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:34.861295 12453 raft_consensus.cc:729] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31ecc2825f0413191dcf9667e7ac9f0, State: Initialized, Role: FOLLOWER
I20260810 14:44:34.861291 12455 raft_consensus.cc:348] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8173e8c9abb347a4b6c75e9b76d5e71e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42981 } } peers { permanent_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 43441 } } peers { permanent_uuid: "194fc540ade146c38a8e4893104f1f5d" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39641 } }
I20260810 14:44:34.861549 12455 raft_consensus.cc:374] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:34.861604 12455 raft_consensus.cc:729] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8173e8c9abb347a4b6c75e9b76d5e71e, State: Initialized, Role: FOLLOWER
I20260810 14:44:34.861874 12453 consensus_queue.cc:260] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8173e8c9abb347a4b6c75e9b76d5e71e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42981 } } peers { permanent_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 43441 } } peers { permanent_uuid: "194fc540ade146c38a8e4893104f1f5d" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39641 } }
I20260810 14:44:34.862064 12455 consensus_queue.cc:260] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8173e8c9abb347a4b6c75e9b76d5e71e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42981 } } peers { permanent_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 43441 } } peers { permanent_uuid: "194fc540ade146c38a8e4893104f1f5d" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39641 } }
I20260810 14:44:34.862671 12231 heartbeater.cc:503] Master 127.15.84.62:40453 was elected leader, sending a full tablet report...
I20260810 14:44:34.862982 12453 ts_tablet_manager.cc:1428] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 14:44:34.863035 12455 ts_tablet_manager.cc:1428] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 14:44:34.863374 12036 heartbeater.cc:503] Master 127.15.84.62:40453 was elected leader, sending a full tablet report...
I20260810 14:44:34.865484 12454 raft_consensus.cc:348] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8173e8c9abb347a4b6c75e9b76d5e71e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42981 } } peers { permanent_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 43441 } } peers { permanent_uuid: "194fc540ade146c38a8e4893104f1f5d" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39641 } }
I20260810 14:44:34.865777 12454 raft_consensus.cc:374] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:34.865834 12454 raft_consensus.cc:729] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 194fc540ade146c38a8e4893104f1f5d, State: Initialized, Role: FOLLOWER
I20260810 14:44:34.866330 12454 consensus_queue.cc:260] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8173e8c9abb347a4b6c75e9b76d5e71e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42981 } } peers { permanent_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 43441 } } peers { permanent_uuid: "194fc540ade146c38a8e4893104f1f5d" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39641 } }
I20260810 14:44:34.867053 12429 heartbeater.cc:503] Master 127.15.84.62:40453 was elected leader, sending a full tablet report...
I20260810 14:44:34.867497 12454 ts_tablet_manager.cc:1428] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260810 14:44:35.031468 12431 tablet.cc:2369] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:44:35.057559 12232 tablet.cc:2369] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:44:35.082980 12038 tablet.cc:2369] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:44:35.116554 12459 raft_consensus.cc:482] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:44:35.116796 12459 raft_consensus.cc:504] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8173e8c9abb347a4b6c75e9b76d5e71e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42981 } } peers { permanent_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 43441 } } peers { permanent_uuid: "194fc540ade146c38a8e4893104f1f5d" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39641 } }
I20260810 14:44:35.118009 12459 leader_election.cc:288] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8173e8c9abb347a4b6c75e9b76d5e71e (127.15.84.1:42981), 194fc540ade146c38a8e4893104f1f5d (127.15.84.3:39641)
I20260810 14:44:35.122164 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:35.118531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:56923 (local address 127.15.84.1:42981)
0810 14:44:35.118665 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:35.118669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:35.118823 (+ 154us) server_negotiation.cc:407] Connection header received
0810 14:44:35.118969 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:35.118971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:35.119026 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:35.119090 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:35.119411 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.119912 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.121173 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.121584 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.121682 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:35.121821 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:35.121969 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 14:44:35.122050 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 14:44:35.122188 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:35.118280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:42981 (local address 127.15.84.2:56923)
0810 14:44:35.118727 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:35.118739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:35.118859 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:35.119119 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:35.119122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:35.119165 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:35.119295 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:35.119300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.120019 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.120023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:35.121031 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:35.121042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.121730 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.121734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:35.121848 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:35.121873 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:44:35.121942 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 14:44:35.122021 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":177,"spinlock_wait_cycles":768,"thread_start_us":82,"threads_started":1}
I20260810 14:44:35.122258 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:35.118280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:39641 (local address 127.15.84.2:49037)
0810 14:44:35.118724 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:35.118738 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:35.118875 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:35.119114 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:35.119117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:35.119156 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:35.119282 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:35.119288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.120000 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.120003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:35.121019 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:35.121030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.121800 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.121804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:35.121939 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:35.121957 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 14:44:35.122027 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 14:44:35.122101 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":69,"spinlock_wait_cycles":22016,"thread_start_us":92,"threads_started":1}
I20260810 14:44:35.122242 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:35.118436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:49037 (local address 127.15.84.3:39641)
0810 14:44:35.118588 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:35.118593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:35.118849 (+ 256us) server_negotiation.cc:407] Connection header received
0810 14:44:35.118984 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:35.118988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:35.119032 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:35.119095 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:35.119390 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.119895 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.121169 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.121615 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.121713 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:35.121834 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:35.122058 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 14:44:35.122129 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:44:35.122824 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f095b5adb2941d0affdf0ada9248e10" candidate_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8173e8c9abb347a4b6c75e9b76d5e71e" is_pre_election: true
I20260810 14:44:35.122998 12354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f095b5adb2941d0affdf0ada9248e10" candidate_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "194fc540ade146c38a8e4893104f1f5d" is_pre_election: true
I20260810 14:44:35.123152 11961 raft_consensus.cc:2454] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c31ecc2825f0413191dcf9667e7ac9f0 in term 0.
I20260810 14:44:35.123310 12354 raft_consensus.cc:2454] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c31ecc2825f0413191dcf9667e7ac9f0 in term 0.
I20260810 14:44:35.123662 12071 leader_election.cc:302] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 194fc540ade146c38a8e4893104f1f5d, 8173e8c9abb347a4b6c75e9b76d5e71e, c31ecc2825f0413191dcf9667e7ac9f0; no voters:
I20260810 14:44:35.123984 12459 raft_consensus.cc:2781] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:44:35.124073 12459 raft_consensus.cc:482] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:44:35.124127 12459 raft_consensus.cc:3037] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:35.125270 12459 raft_consensus.cc:504] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8173e8c9abb347a4b6c75e9b76d5e71e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42981 } } peers { permanent_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 43441 } } peers { permanent_uuid: "194fc540ade146c38a8e4893104f1f5d" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39641 } }
I20260810 14:44:35.125625 12459 leader_election.cc:288] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [CANDIDATE]: Term 1 election: Requested vote from peers 8173e8c9abb347a4b6c75e9b76d5e71e (127.15.84.1:42981), 194fc540ade146c38a8e4893104f1f5d (127.15.84.3:39641)
I20260810 14:44:35.125985 12354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f095b5adb2941d0affdf0ada9248e10" candidate_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "194fc540ade146c38a8e4893104f1f5d"
I20260810 14:44:35.125978 11961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f095b5adb2941d0affdf0ada9248e10" candidate_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8173e8c9abb347a4b6c75e9b76d5e71e"
I20260810 14:44:35.126116 12354 raft_consensus.cc:3037] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:35.126116 11961 raft_consensus.cc:3037] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:35.127442 11961 raft_consensus.cc:2454] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c31ecc2825f0413191dcf9667e7ac9f0 in term 1.
I20260810 14:44:35.127442 12354 raft_consensus.cc:2454] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c31ecc2825f0413191dcf9667e7ac9f0 in term 1.
I20260810 14:44:35.127800 12071 leader_election.cc:302] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8173e8c9abb347a4b6c75e9b76d5e71e, c31ecc2825f0413191dcf9667e7ac9f0; no voters:
I20260810 14:44:35.127985 12459 raft_consensus.cc:2781] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:35.128278 12459 raft_consensus.cc:686] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [term 1 LEADER]: Becoming Leader. State: Replica: c31ecc2825f0413191dcf9667e7ac9f0, State: Running, Role: LEADER
I20260810 14:44:35.128665 12459 consensus_queue.cc:237] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8173e8c9abb347a4b6c75e9b76d5e71e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42981 } } peers { permanent_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 43441 } } peers { permanent_uuid: "194fc540ade146c38a8e4893104f1f5d" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39641 } }
I20260810 14:44:35.132102 11738 catalog_manager.cc:5495] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c31ecc2825f0413191dcf9667e7ac9f0 (127.15.84.2). New cstate: current_term: 1 leader_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8173e8c9abb347a4b6c75e9b76d5e71e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c31ecc2825f0413191dcf9667e7ac9f0" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 43441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "194fc540ade146c38a8e4893104f1f5d" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39641 } health_report { overall_health: UNKNOWN } } }
I20260810 14:44:35.211390 12448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:35.207693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.15.84.2:43441)
0810 14:44:35.207833 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:35.207837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:35.207852 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:44:35.207947 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:35.207950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:35.208011 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:35.208108 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:44:35.208328 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.208877 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.209870 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.210107 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.210201 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:35.210518 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:44:35.210956 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:44:35.211081 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:35.211210 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 14:44:35.211253 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:44:35.218081 11961 raft_consensus.cc:1261] T 8f095b5adb2941d0affdf0ada9248e10 P 8173e8c9abb347a4b6c75e9b76d5e71e [term 1 FOLLOWER]: Refusing update from remote peer c31ecc2825f0413191dcf9667e7ac9f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:44:35.218081 12354 raft_consensus.cc:1261] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d [term 1 FOLLOWER]: Refusing update from remote peer c31ecc2825f0413191dcf9667e7ac9f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:44:35.219028 12468 consensus_queue.cc:1035] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "194fc540ade146c38a8e4893104f1f5d" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 39641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:44:35.219255 12459 consensus_queue.cc:1035] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8173e8c9abb347a4b6c75e9b76d5e71e" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 42981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:44:35.227792 12471 mvcc.cc:205] Tried to move back new op lower bound from 7316984116084793344 to 7316984115730300928. Current Snapshot: MvccSnapshot[applied={T|T < 7316984116084793344}]
I20260810 14:44:35.236637 11736 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:44:35.236812 11736 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:44:35.239461 11736 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95b5660219e8455abbfbddf08f03ef35: Sending DeleteTablet for 3 replicas of tablet 8f095b5adb2941d0affdf0ada9248e10
I20260810 14:44:35.240335 12137 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f095b5adb2941d0affdf0ada9248e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:35 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:57668
I20260810 14:44:35.240458 11938 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f095b5adb2941d0affdf0ada9248e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:35 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:43962
I20260810 14:44:35.240978 12325 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f095b5adb2941d0affdf0ada9248e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:35 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:39926
I20260810 14:44:35.241544 12493 tablet_replica.cc:330] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d: stopping tablet replica
I20260810 14:44:35.241544 12492 tablet_replica.cc:330] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0: stopping tablet replica
I20260810 14:44:35.241809 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 11834
I20260810 14:44:35.250849 12492 raft_consensus.cc:2229] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:44:35.251394 12492 raft_consensus.cc:2258] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:44:35.252702 12493 raft_consensus.cc:2229] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:44:35.253060 12493 raft_consensus.cc:2258] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:44:35.254678 12492 ts_tablet_manager.cc:1905] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:44:35.256165 12493 ts_tablet_manager.cc:1905] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:44:35.257944 12492 ts_tablet_manager.cc:1918] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:44:35.258206 12492 log.cc:1196] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/8f095b5adb2941d0affdf0ada9248e10
I20260810 14:44:35.258647 12492 ts_tablet_manager.cc:1939] T 8f095b5adb2941d0affdf0ada9248e10 P c31ecc2825f0413191dcf9667e7ac9f0: Deleting consensus metadata
I20260810 14:44:35.259029 12493 ts_tablet_manager.cc:1918] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:44:35.259141 12493 log.cc:1196] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/8f095b5adb2941d0affdf0ada9248e10
I20260810 14:44:35.259438 12493 ts_tablet_manager.cc:1939] T 8f095b5adb2941d0affdf0ada9248e10 P 194fc540ade146c38a8e4893104f1f5d: Deleting consensus metadata
I20260810 14:44:35.260495 11725 catalog_manager.cc:4841] TS c31ecc2825f0413191dcf9667e7ac9f0 (127.15.84.2:43441): tablet 8f095b5adb2941d0affdf0ada9248e10 (table test-table [id=eef9d48a572848668637103943fbee57]) successfully deleted
I20260810 14:44:35.260684 11724 catalog_manager.cc:4841] TS 194fc540ade146c38a8e4893104f1f5d (127.15.84.3:39641): tablet 8f095b5adb2941d0affdf0ada9248e10 (table test-table [id=eef9d48a572848668637103943fbee57]) successfully deleted
W20260810 14:44:35.263062 11721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:42981 (error 108)
I20260810 14:44:35.264078 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 12046
I20260810 14:44:35.264300 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:35.263971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:42981 (local address 127.0.0.1:43968)
0810 14:44:35.264131 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:35.264206 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 14:44:35.276505 11721 catalog_manager.cc:4584] TS 8173e8c9abb347a4b6c75e9b76d5e71e (127.15.84.1:42981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f095b5adb2941d0affdf0ada9248e10: Network error: Client connection negotiation failed: client connection to 127.15.84.1:42981: connect: Connection refused (error 111)
I20260810 14:44:35.283097 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 12240
I20260810 14:44:35.299381 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 11697
2026-08-10T14:44:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2767 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T14:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:35Z Disabled control of system clock
I20260810 14:44:35.341562 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:42563
--webserver_interface=127.15.84.62
--webserver_port=0
--builtin_ntp_servers=127.15.84.20:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:42563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:44:35.458662 12502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:35.458966 12502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:35.459009 12502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:35.463182 12502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:35.463274 12502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:35.463287 12502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:35.463304 12502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:35.463318 12502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:35.468497 12502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:42563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:42563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:35.469733 12502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:35.471227 12502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:35.480500 12510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:35.480489 12514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:35.480494 12509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:35.480880 12502 server_base.cc:1034] running on GCE node
I20260810 14:44:35.481555 12502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:35.482817 12502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:35.484007 12502 hybrid_clock.cc:648] HybridClock initialized: now 1786373075483982 us; error 43 us; skew 500 ppm
I20260810 14:44:35.486132 12502 webserver.cc:460] Webserver started at http://127.15.84.62:41391/ using document root <none> and password file <none>
I20260810 14:44:35.486764 12502 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:35.486840 12502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:35.487061 12502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:35.489391 12502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "9440a9f512cc4fc69da1dd2787b527ed"
format_stamp: "Formatted at 2026-08-10 14:44:35 on dist-test-slave-rmhg"
server_key: "510ceead34ecdf8f3171a48523803aed"
server_key_iv: "4d1b0af90428eb377b6ef9d4ac3bf8f2"
server_key_version: "encryptionkey@0"
I20260810 14:44:35.490042 12502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "9440a9f512cc4fc69da1dd2787b527ed"
format_stamp: "Formatted at 2026-08-10 14:44:35 on dist-test-slave-rmhg"
server_key: "510ceead34ecdf8f3171a48523803aed"
server_key_iv: "4d1b0af90428eb377b6ef9d4ac3bf8f2"
server_key_version: "encryptionkey@0"
I20260810 14:44:35.494309 12502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:44:35.497336 12525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:35.498876 12502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:44:35.499027 12502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "9440a9f512cc4fc69da1dd2787b527ed"
format_stamp: "Formatted at 2026-08-10 14:44:35 on dist-test-slave-rmhg"
server_key: "510ceead34ecdf8f3171a48523803aed"
server_key_iv: "4d1b0af90428eb377b6ef9d4ac3bf8f2"
server_key_version: "encryptionkey@0"
I20260810 14:44:35.499168 12502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:35.512763 12502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:35.513617 12502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:35.513818 12502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:35.521243 12502 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:42563
I20260810 14:44:35.521273 12603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:42563 every 8 connection(s)
I20260810 14:44:35.522413 12502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:35.525875 12604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:35.528728 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 12502
I20260810 14:44:35.528887 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:35.529273 15696 external_mini_cluster.cc:1428] Setting key 7b26c4871ec6f5a51b5b8eaf09aa10c7
I20260810 14:44:35.532982 12604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed: Bootstrap starting.
I20260810 14:44:35.535528 12604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:35.536357 12604 log.cc:824] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:35.538477 12604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed: No bootstrap required, opened a new log
I20260810 14:44:35.539583 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:35.531043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.15.84.62:42563)
0810 14:44:35.531890 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:35.531911 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:35.531950 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:44:35.532640 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:35.532658 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:35.533108 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:35.533539 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:35.534036 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.535086 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.536095 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.536649 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.536811 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:35.537688 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:35.537694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:35.537709 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:35.538138 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:35.538203 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:35.538225 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:35.538317 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:35.538480 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 14:44:35.538816 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":180,"threads_started":1}
I20260810 14:44:35.542233 12604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9440a9f512cc4fc69da1dd2787b527ed" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42563 } }
I20260810 14:44:35.542486 12604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:35.542547 12604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9440a9f512cc4fc69da1dd2787b527ed, State: Initialized, Role: FOLLOWER
I20260810 14:44:35.542984 12604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9440a9f512cc4fc69da1dd2787b527ed" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42563 } }
I20260810 14:44:35.543123 12604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:35.543191 12604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:35.543301 12604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:35.544312 12604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9440a9f512cc4fc69da1dd2787b527ed" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42563 } }
I20260810 14:44:35.544694 12604 leader_election.cc:302] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9440a9f512cc4fc69da1dd2787b527ed; no voters:
I20260810 14:44:35.545018 12604 leader_election.cc:288] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:35.545197 12609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:35.545435 12609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [term 1 LEADER]: Becoming Leader. State: Replica: 9440a9f512cc4fc69da1dd2787b527ed, State: Running, Role: LEADER
I20260810 14:44:35.545766 12609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9440a9f512cc4fc69da1dd2787b527ed" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42563 } }
I20260810 14:44:35.546353 12604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:35.547704 12610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9440a9f512cc4fc69da1dd2787b527ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9440a9f512cc4fc69da1dd2787b527ed" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42563 } } }
I20260810 14:44:35.547794 12610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:35.547596 12614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9440a9f512cc4fc69da1dd2787b527ed. Latest consensus state: current_term: 1 leader_uuid: "9440a9f512cc4fc69da1dd2787b527ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9440a9f512cc4fc69da1dd2787b527ed" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42563 } } }
I20260810 14:44:35.548053 12614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:35.548247 12627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:35.551246 12627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:35.557449 12627 catalog_manager.cc:1347] Generated new cluster ID: f13d90abcc4b40dab7f24ae684d4b00a
I20260810 14:44:35.557547 12627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:35.570032 12627 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:35.571462 12627 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:35.586766 12627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed: Generated new TSK 0
I20260810 14:44:35.587821 12627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:35.591642 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:42563
--builtin_ntp_servers=127.15.84.20:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:44:35.716177 12643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:35.716506 12643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:35.716567 12643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:35.720800 12643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:35.720882 12643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:35.720958 12643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:44:35.726490 12643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:42563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:35.727790 12643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:35.729383 12643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:35.739632 12650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:35.739548 12651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:35.739549 12655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:35.739900 12643 server_base.cc:1034] running on GCE node
I20260810 14:44:35.740556 12643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:35.741326 12643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:35.742493 12643 hybrid_clock.cc:648] HybridClock initialized: now 1786373075742468 us; error 50 us; skew 500 ppm
I20260810 14:44:35.744530 12643 webserver.cc:460] Webserver started at http://127.15.84.1:45533/ using document root <none> and password file <none>
I20260810 14:44:35.745127 12643 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:35.745241 12643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:35.745460 12643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:35.747803 12643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "62dd20faf1fa4f76b20695746b7a2bcc"
format_stamp: "Formatted at 2026-08-10 14:44:35 on dist-test-slave-rmhg"
server_key: "072ff646c01e3e2d637671b12c6e864a"
server_key_iv: "6bbf3f63fcbf08b8413d9c0faa807463"
server_key_version: "encryptionkey@0"
I20260810 14:44:35.748401 12643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "62dd20faf1fa4f76b20695746b7a2bcc"
format_stamp: "Formatted at 2026-08-10 14:44:35 on dist-test-slave-rmhg"
server_key: "072ff646c01e3e2d637671b12c6e864a"
server_key_iv: "6bbf3f63fcbf08b8413d9c0faa807463"
server_key_version: "encryptionkey@0"
I20260810 14:44:35.752416 12643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:44:35.755110 12663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:35.756449 12643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:44:35.756603 12643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "62dd20faf1fa4f76b20695746b7a2bcc"
format_stamp: "Formatted at 2026-08-10 14:44:35 on dist-test-slave-rmhg"
server_key: "072ff646c01e3e2d637671b12c6e864a"
server_key_iv: "6bbf3f63fcbf08b8413d9c0faa807463"
server_key_version: "encryptionkey@0"
I20260810 14:44:35.756727 12643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:35.771420 12643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:35.772192 12643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:35.772385 12643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:35.773043 12643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:35.774111 12643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:35.774163 12643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:35.774219 12643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:35.774248 12643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:35.783263 12643 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:35129
I20260810 14:44:35.783309 12846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:35129 every 8 connection(s)
I20260810 14:44:35.784325 12643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:44:35.789814 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 12643
I20260810 14:44:35.789973 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:44:35.790310 15696 external_mini_cluster.cc:1428] Setting key 2d05dc6cea341407495c5b9b0644ac60
I20260810 14:44:35.792618 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:42563
--builtin_ntp_servers=127.15.84.20:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:44:35.797313 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:35.788533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:58557 (local address 127.15.84.62:42563)
0810 14:44:35.788712 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:35.788717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:35.790093 (+ 1376us) server_negotiation.cc:407] Connection header received
0810 14:44:35.791755 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:35.791759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:35.791812 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:35.791902 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:35.793226 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.793771 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.795188 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.795422 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.795515 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:35.796808 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:35.796811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:35.796813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:35.796829 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:35.796851 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:35.796859 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:35.796919 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:35.797085 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 14:44:35.797181 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 14:44:35.798616 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:35.789046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:42563 (local address 127.15.84.1:58557)
0810 14:44:35.789897 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:35.789963 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:35.791513 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:35.792132 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:35.792143 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:35.792267 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:35.793054 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:35.793075 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.793905 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.793909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:35.795047 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:35.795058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.795584 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.795589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:35.795776 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:35.796479 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:44:35.796481 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:35.796539 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:44:35.796628 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:35.796938 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:35.796947 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:35.796954 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:44:35.797082 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 14:44:35.797435 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":159,"threads_started":1}
I20260810 14:44:35.799957 12847 heartbeater.cc:347] Connected to a master server at 127.15.84.62:42563
I20260810 14:44:35.800251 12847 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:35.800969 12847 heartbeater.cc:511] Master 127.15.84.62:42563 requested a full tablet report, sending...
I20260810 14:44:35.802719 12548 ts_manager.cc:194] Registered new tserver with Master: 62dd20faf1fa4f76b20695746b7a2bcc (127.15.84.1:35129)
I20260810 14:44:35.803917 12548 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.84.1:58557
W20260810 14:44:35.913679 12856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:35.913949 12856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:35.913990 12856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:35.918128 12856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:35.918206 12856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:35.918265 12856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:44:35.923231 12856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:42563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:35.924506 12856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:35.926048 12856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:35.935967 12865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:35.936029 12856 server_base.cc:1034] running on GCE node
W20260810 14:44:35.935961 12866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:35.936262 12868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:35.936678 12856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:35.937386 12856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:35.938581 12856 hybrid_clock.cc:648] HybridClock initialized: now 1786373075938555 us; error 35 us; skew 500 ppm
I20260810 14:44:35.940528 12856 webserver.cc:460] Webserver started at http://127.15.84.2:43427/ using document root <none> and password file <none>
I20260810 14:44:35.941119 12856 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:35.941227 12856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:35.941457 12856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:35.943599 12856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "a61d602ebadc4044b16e3b9b67876d6e"
format_stamp: "Formatted at 2026-08-10 14:44:35 on dist-test-slave-rmhg"
server_key: "4e11ac35972d0b6a119240a983ac468e"
server_key_iv: "28c78ec3c64e476ef3c88b7dcbddf22a"
server_key_version: "encryptionkey@0"
I20260810 14:44:35.944149 12856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "a61d602ebadc4044b16e3b9b67876d6e"
format_stamp: "Formatted at 2026-08-10 14:44:35 on dist-test-slave-rmhg"
server_key: "4e11ac35972d0b6a119240a983ac468e"
server_key_iv: "28c78ec3c64e476ef3c88b7dcbddf22a"
server_key_version: "encryptionkey@0"
I20260810 14:44:35.948210 12856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:44:35.950903 12876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:35.952198 12856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:44:35.952316 12856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "a61d602ebadc4044b16e3b9b67876d6e"
format_stamp: "Formatted at 2026-08-10 14:44:35 on dist-test-slave-rmhg"
server_key: "4e11ac35972d0b6a119240a983ac468e"
server_key_iv: "28c78ec3c64e476ef3c88b7dcbddf22a"
server_key_version: "encryptionkey@0"
I20260810 14:44:35.952416 12856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:35.968264 12856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:35.968997 12856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:35.969190 12856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:35.969795 12856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:35.970770 12856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:35.970818 12856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:35.970862 12856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:35.970892 12856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:35.979501 12856 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:42133
I20260810 14:44:35.979533 13051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:42133 every 8 connection(s)
I20260810 14:44:35.980461 12856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:44:35.981253 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 12856
I20260810 14:44:35.981374 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:44:35.981724 15696 external_mini_cluster.cc:1428] Setting key 643b861fbd0721403bb86a83a9866ca4
I20260810 14:44:35.983777 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:42563
--builtin_ntp_servers=127.15.84.20:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:44:35.990329 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:35.982530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:35871 (local address 127.15.84.62:42563)
0810 14:44:35.982662 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:35.982666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:35.983811 (+ 1145us) server_negotiation.cc:407] Connection header received
0810 14:44:35.985287 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:35.985291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:35.985349 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:35.985441 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:35.986368 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.986903 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.988147 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.988441 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.988538 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:35.989798 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:35.989803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:35.989806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:35.989825 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:35.989852 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:35.989865 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:35.989942 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:35.990076 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 14:44:35.990152 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:44:35.991227 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:35.982881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:42563 (local address 127.15.84.2:35871)
0810 14:44:35.983618 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:35.983684 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:35.985044 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:35.985594 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:35.985601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:35.985705 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:35.986242 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:35.986255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.987050 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.987054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:35.988016 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:35.988026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:35.988611 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:35.988614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:35.988787 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:35.989504 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:44:35.989505 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:35.989551 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:44:35.989621 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:35.989951 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:35.989957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:35.989961 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:44:35.990057 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 14:44:35.990356 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":118,"threads_started":1}
I20260810 14:44:35.992547 13052 heartbeater.cc:347] Connected to a master server at 127.15.84.62:42563
I20260810 14:44:35.992815 13052 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:35.993435 13052 heartbeater.cc:511] Master 127.15.84.62:42563 requested a full tablet report, sending...
I20260810 14:44:35.994597 12548 ts_manager.cc:194] Registered new tserver with Master: a61d602ebadc4044b16e3b9b67876d6e (127.15.84.2:42133)
I20260810 14:44:35.995186 12548 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.84.2:35871
W20260810 14:44:36.102025 13058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:36.102311 13058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:36.102350 13058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:36.106560 13058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:36.106634 13058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:36.106693 13058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:44:36.111665 13058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:42563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:36.112844 13058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:36.114316 13058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:36.124166 13069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:36.124193 13067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:36.124220 13065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:36.124608 13058 server_base.cc:1034] running on GCE node
I20260810 14:44:36.125069 13058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:36.125801 13058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:36.126997 13058 hybrid_clock.cc:648] HybridClock initialized: now 1786373076126972 us; error 35 us; skew 500 ppm
I20260810 14:44:36.128968 13058 webserver.cc:460] Webserver started at http://127.15.84.3:45373/ using document root <none> and password file <none>
I20260810 14:44:36.129626 13058 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:36.129702 13058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:36.129936 13058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:36.132112 13058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "24e371f6b4724a82a448e05f5dbbd23b"
format_stamp: "Formatted at 2026-08-10 14:44:36 on dist-test-slave-rmhg"
server_key: "dc8dc9f6885a87c520f8b532e8932613"
server_key_iv: "1763b0a9b50c7eb40334cb8cd63fac5c"
server_key_version: "encryptionkey@0"
I20260810 14:44:36.132686 13058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "24e371f6b4724a82a448e05f5dbbd23b"
format_stamp: "Formatted at 2026-08-10 14:44:36 on dist-test-slave-rmhg"
server_key: "dc8dc9f6885a87c520f8b532e8932613"
server_key_iv: "1763b0a9b50c7eb40334cb8cd63fac5c"
server_key_version: "encryptionkey@0"
I20260810 14:44:36.136684 13058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:44:36.139513 13075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:36.140956 13058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 14:44:36.141091 13058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "24e371f6b4724a82a448e05f5dbbd23b"
format_stamp: "Formatted at 2026-08-10 14:44:36 on dist-test-slave-rmhg"
server_key: "dc8dc9f6885a87c520f8b532e8932613"
server_key_iv: "1763b0a9b50c7eb40334cb8cd63fac5c"
server_key_version: "encryptionkey@0"
I20260810 14:44:36.141273 13058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:36.168133 13058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:36.168977 13058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:36.169219 13058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:36.169862 13058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:36.170897 13058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:36.170948 13058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:36.171003 13058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:36.171033 13058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:36.179898 13058 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:36013
I20260810 14:44:36.179924 13255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:36013 every 8 connection(s)
I20260810 14:44:36.180881 13058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:44:36.183352 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 13058
I20260810 14:44:36.183470 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:44:36.183765 15696 external_mini_cluster.cc:1428] Setting key f6a7e3dca270adef0ad29f18c2b90c39
I20260810 14:44:36.190709 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.182872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:39643 (local address 127.15.84.62:42563)
0810 14:44:36.183053 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:36.183058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:36.184392 (+ 1334us) server_negotiation.cc:407] Connection header received
0810 14:44:36.186111 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:36.186113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:36.186163 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:36.186229 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:36.187154 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.187695 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.188815 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.189032 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.189101 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:36.190201 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:36.190204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:36.190207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:36.190221 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:36.190244 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:36.190252 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:36.190314 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:36.190515 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 14:44:36.190577 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 14:44:36.191732 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.183377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:42563 (local address 127.15.84.3:39643)
0810 14:44:36.184219 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:36.184272 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:36.185891 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:36.186381 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:36.186388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:36.186480 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:36.187004 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:36.187017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.187798 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.187801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:36.188720 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:36.188730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.189166 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.189170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:36.189353 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:36.189923 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:44:36.189925 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:36.189970 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:44:36.190042 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:36.190366 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:36.190377 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:36.190383 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:44:36.190540 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 14:44:36.190814 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":144,"threads_started":1}
I20260810 14:44:36.193015 13258 heartbeater.cc:347] Connected to a master server at 127.15.84.62:42563
I20260810 14:44:36.193336 13258 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:36.193912 13258 heartbeater.cc:511] Master 127.15.84.62:42563 requested a full tablet report, sending...
I20260810 14:44:36.195080 12548 ts_manager.cc:194] Registered new tserver with Master: 24e371f6b4724a82a448e05f5dbbd23b (127.15.84.3:36013)
I20260810 14:44:36.195662 12548 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.84.3:39643
I20260810 14:44:36.197897 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:44:36.204048 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.199867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.15.84.62:42563)
0810 14:44:36.200006 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:36.200011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:36.200177 (+ 166us) server_negotiation.cc:407] Connection header received
0810 14:44:36.200317 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:36.200320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:36.200366 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:36.200426 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:36.200708 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.201293 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.202126 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.202360 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.202453 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:36.203436 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:36.203441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:36.203443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:36.203457 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:36.203477 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:36.203485 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:36.203598 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:36.203832 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 14:44:36.203880 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:44:36.207257 12548 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:44:36.210433 12548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:44:36.225212 13278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.219333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:42133 (local address 127.0.0.1:56764)
0810 14:44:36.219774 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:36.219826 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:36.219951 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:36.221102 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:36.221108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:36.221133 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:36.221345 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:36.221353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.222545 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.222549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:36.223711 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:36.223723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.224733 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.224736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:36.224824 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:36.224859 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 14:44:36.224935 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 14:44:36.225030 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":102,"threads_started":1}
I20260810 14:44:36.225301 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.220075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56764 (local address 127.15.84.2:42133)
0810 14:44:36.220556 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:36.220568 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:36.220595 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:44:36.220692 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:36.220698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:36.220921 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:36.221087 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:36.221480 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.222443 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.223972 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.224612 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.224717 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:36.224934 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:36.224988 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 14:44:36.225092 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":126,"threads_started":1}
I20260810 14:44:36.225889 13281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.219653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:36013 (local address 127.0.0.1:38146)
0810 14:44:36.221274 (+ 1621us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:36.221287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:36.221383 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:36.221821 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:36.221824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:36.221842 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:36.221919 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:36.221924 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.223201 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.223204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:36.224396 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:36.224408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.225501 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.225505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:36.225596 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:36.225618 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:44:36.225679 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 14:44:36.225753 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1446,"mutex_wait_us":29,"thread_start_us":95,"threads_started":1}
I20260810 14:44:36.225986 13285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.220288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.15.84.3:36013)
0810 14:44:36.220718 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:36.220729 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:36.221396 (+ 667us) server_negotiation.cc:407] Connection header received
0810 14:44:36.221479 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:36.221484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:36.221680 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:36.221838 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:36.222012 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.223084 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.224539 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.225375 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.225482 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:36.225689 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:36.225739 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 14:44:36.225834 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":137,"threads_started":1}
I20260810 14:44:36.227762 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.219446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:35129 (local address 127.0.0.1:38272)
0810 14:44:36.220883 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:36.220897 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:36.220988 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:36.221477 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:36.221480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:36.221493 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:36.221581 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:36.221587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.223002 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.223005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:36.224399 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:36.224408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.227404 (+ 2996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.227407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:36.227476 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:36.227494 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 14:44:36.227558 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 14:44:36.227630 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1375,"mutex_wait_us":53,"thread_start_us":106,"threads_started":1}
I20260810 14:44:36.227878 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.220273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38272 (local address 127.15.84.1:35129)
0810 14:44:36.220718 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:36.220730 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:36.220996 (+ 266us) server_negotiation.cc:407] Connection header received
0810 14:44:36.221082 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:36.221088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:36.221315 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:36.221459 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:36.221687 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.222847 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.225262 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.227055 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.227158 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:36.227347 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:36.227620 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 14:44:36.227729 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":133,"threads_started":1}
I20260810 14:44:36.228850 12955 tablet_service.cc:1467] Processing CreateTablet for tablet c969a805890441969e3681de4b0f2e3d (DEFAULT_TABLE table=test-table [id=4e6ccae09040401ba56e5a200a806fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:36.228850 13159 tablet_service.cc:1467] Processing CreateTablet for tablet c969a805890441969e3681de4b0f2e3d (DEFAULT_TABLE table=test-table [id=4e6ccae09040401ba56e5a200a806fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:36.230125 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c969a805890441969e3681de4b0f2e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:36.230131 13159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c969a805890441969e3681de4b0f2e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:36.230490 12741 tablet_service.cc:1467] Processing CreateTablet for tablet c969a805890441969e3681de4b0f2e3d (DEFAULT_TABLE table=test-table [id=4e6ccae09040401ba56e5a200a806fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:36.231599 12741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c969a805890441969e3681de4b0f2e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:36.237649 13286 tablet_bootstrap.cc:492] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc: Bootstrap starting.
I20260810 14:44:36.238916 13288 tablet_bootstrap.cc:492] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e: Bootstrap starting.
I20260810 14:44:36.239917 13286 tablet_bootstrap.cc:654] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:36.240655 13286 log.cc:824] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:36.240739 13288 tablet_bootstrap.cc:654] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:36.241411 13287 tablet_bootstrap.cc:492] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b: Bootstrap starting.
I20260810 14:44:36.241583 13288 log.cc:824] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:36.242985 13286 tablet_bootstrap.cc:492] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc: No bootstrap required, opened a new log
I20260810 14:44:36.243207 13286 ts_tablet_manager.cc:1397] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:44:36.243693 13287 tablet_bootstrap.cc:654] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:36.243767 13288 tablet_bootstrap.cc:492] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e: No bootstrap required, opened a new log
I20260810 14:44:36.243963 13288 ts_tablet_manager.cc:1397] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 14:44:36.244632 13287 log.cc:824] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:36.246989 13287 tablet_bootstrap.cc:492] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b: No bootstrap required, opened a new log
I20260810 14:44:36.247222 13287 ts_tablet_manager.cc:1397] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:44:36.247095 13286 raft_consensus.cc:348] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d602ebadc4044b16e3b9b67876d6e" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 42133 } } peers { permanent_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 35129 } } peers { permanent_uuid: "24e371f6b4724a82a448e05f5dbbd23b" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 36013 } }
I20260810 14:44:36.247417 13286 raft_consensus.cc:374] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:36.247475 13286 raft_consensus.cc:729] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62dd20faf1fa4f76b20695746b7a2bcc, State: Initialized, Role: FOLLOWER
I20260810 14:44:36.247958 13286 consensus_queue.cc:260] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d602ebadc4044b16e3b9b67876d6e" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 42133 } } peers { permanent_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 35129 } } peers { permanent_uuid: "24e371f6b4724a82a448e05f5dbbd23b" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 36013 } }
I20260810 14:44:36.248697 12847 heartbeater.cc:503] Master 127.15.84.62:42563 was elected leader, sending a full tablet report...
I20260810 14:44:36.249058 13286 ts_tablet_manager.cc:1428] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:44:36.249810 13288 raft_consensus.cc:348] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d602ebadc4044b16e3b9b67876d6e" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 42133 } } peers { permanent_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 35129 } } peers { permanent_uuid: "24e371f6b4724a82a448e05f5dbbd23b" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 36013 } }
I20260810 14:44:36.250106 13288 raft_consensus.cc:374] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:36.250175 13288 raft_consensus.cc:729] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61d602ebadc4044b16e3b9b67876d6e, State: Initialized, Role: FOLLOWER
I20260810 14:44:36.250654 13288 consensus_queue.cc:260] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d602ebadc4044b16e3b9b67876d6e" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 42133 } } peers { permanent_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 35129 } } peers { permanent_uuid: "24e371f6b4724a82a448e05f5dbbd23b" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 36013 } }
I20260810 14:44:36.251432 13052 heartbeater.cc:503] Master 127.15.84.62:42563 was elected leader, sending a full tablet report...
I20260810 14:44:36.251767 13288 ts_tablet_manager.cc:1428] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 14:44:36.252591 13287 raft_consensus.cc:348] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d602ebadc4044b16e3b9b67876d6e" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 42133 } } peers { permanent_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 35129 } } peers { permanent_uuid: "24e371f6b4724a82a448e05f5dbbd23b" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 36013 } }
I20260810 14:44:36.252895 13287 raft_consensus.cc:374] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:36.252961 13287 raft_consensus.cc:729] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e371f6b4724a82a448e05f5dbbd23b, State: Initialized, Role: FOLLOWER
I20260810 14:44:36.253480 13287 consensus_queue.cc:260] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d602ebadc4044b16e3b9b67876d6e" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 42133 } } peers { permanent_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 35129 } } peers { permanent_uuid: "24e371f6b4724a82a448e05f5dbbd23b" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 36013 } }
I20260810 14:44:36.254418 13287 ts_tablet_manager.cc:1428] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 14:44:36.254794 13258 heartbeater.cc:503] Master 127.15.84.62:42563 was elected leader, sending a full tablet report...
W20260810 14:44:36.287784 12848 tablet.cc:2369] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:44:36.417670 13293 raft_consensus.cc:482] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:44:36.417929 13293 raft_consensus.cc:504] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d602ebadc4044b16e3b9b67876d6e" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 42133 } } peers { permanent_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 35129 } } peers { permanent_uuid: "24e371f6b4724a82a448e05f5dbbd23b" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 36013 } }
I20260810 14:44:36.419195 13293 leader_election.cc:288] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a61d602ebadc4044b16e3b9b67876d6e (127.15.84.2:42133), 24e371f6b4724a82a448e05f5dbbd23b (127.15.84.3:36013)
I20260810 14:44:36.423269 13285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.419672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:55961 (local address 127.15.84.3:36013)
0810 14:44:36.419796 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:36.419801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:36.419959 (+ 158us) server_negotiation.cc:407] Connection header received
0810 14:44:36.420108 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:36.420110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:36.420152 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:36.420214 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:36.420509 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.421032 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.422247 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.422652 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.422748 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:36.422816 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:36.423049 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 14:44:36.423127 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:44:36.423280 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.419607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:59539 (local address 127.15.84.2:42133)
0810 14:44:36.419751 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:36.419755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:36.419993 (+ 238us) server_negotiation.cc:407] Connection header received
0810 14:44:36.420137 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:36.420140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:36.420184 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:36.420248 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:36.420535 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.421031 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.422249 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.422652 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.422751 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:36.422816 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:36.423067 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 14:44:36.423150 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:44:36.423298 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.419524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:36013 (local address 127.15.84.1:55961)
0810 14:44:36.419865 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:36.419877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:36.419993 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:36.420230 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:36.420233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:36.420249 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:36.420395 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:36.420401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.421140 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.421144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:36.422115 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:36.422125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.422770 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.422773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:36.422919 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:36.422945 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:44:36.423013 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 14:44:36.423109 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":62,"thread_start_us":103,"threads_started":1}
I20260810 14:44:36.423306 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.419568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:42133 (local address 127.15.84.1:59539)
0810 14:44:36.419887 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:36.419899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:36.420006 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:36.420271 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:36.420274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:36.420301 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:36.420414 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:36.420419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.421145 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.421174 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:36.422116 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:36.422127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.422790 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.422793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:36.422932 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:36.422951 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 14:44:36.423037 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 14:44:36.423116 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":95,"spinlock_wait_cycles":768,"thread_start_us":88,"threads_started":1}
I20260810 14:44:36.424259 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c969a805890441969e3681de4b0f2e3d" candidate_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a61d602ebadc4044b16e3b9b67876d6e" is_pre_election: true
I20260810 14:44:36.424259 13189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c969a805890441969e3681de4b0f2e3d" candidate_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e371f6b4724a82a448e05f5dbbd23b" is_pre_election: true
I20260810 14:44:36.424614 12984 raft_consensus.cc:2454] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62dd20faf1fa4f76b20695746b7a2bcc in term 0.
I20260810 14:44:36.424613 13189 raft_consensus.cc:2454] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62dd20faf1fa4f76b20695746b7a2bcc in term 0.
I20260810 14:44:36.425192 12666 leader_election.cc:302] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 24e371f6b4724a82a448e05f5dbbd23b, 62dd20faf1fa4f76b20695746b7a2bcc, a61d602ebadc4044b16e3b9b67876d6e; no voters:
I20260810 14:44:36.425482 13293 raft_consensus.cc:2781] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:44:36.425590 13293 raft_consensus.cc:482] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:44:36.425634 13293 raft_consensus.cc:3037] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:36.426728 13293 raft_consensus.cc:504] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d602ebadc4044b16e3b9b67876d6e" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 42133 } } peers { permanent_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 35129 } } peers { permanent_uuid: "24e371f6b4724a82a448e05f5dbbd23b" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 36013 } }
I20260810 14:44:36.427117 13293 leader_election.cc:288] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [CANDIDATE]: Term 1 election: Requested vote from peers a61d602ebadc4044b16e3b9b67876d6e (127.15.84.2:42133), 24e371f6b4724a82a448e05f5dbbd23b (127.15.84.3:36013)
I20260810 14:44:36.427541 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c969a805890441969e3681de4b0f2e3d" candidate_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a61d602ebadc4044b16e3b9b67876d6e"
I20260810 14:44:36.427588 13189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c969a805890441969e3681de4b0f2e3d" candidate_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e371f6b4724a82a448e05f5dbbd23b"
I20260810 14:44:36.427675 12984 raft_consensus.cc:3037] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:36.427734 13189 raft_consensus.cc:3037] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:36.428812 12984 raft_consensus.cc:2454] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62dd20faf1fa4f76b20695746b7a2bcc in term 1.
I20260810 14:44:36.428860 13189 raft_consensus.cc:2454] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62dd20faf1fa4f76b20695746b7a2bcc in term 1.
I20260810 14:44:36.429239 12666 leader_election.cc:302] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 24e371f6b4724a82a448e05f5dbbd23b, 62dd20faf1fa4f76b20695746b7a2bcc, a61d602ebadc4044b16e3b9b67876d6e; no voters:
I20260810 14:44:36.429443 13293 raft_consensus.cc:2781] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:36.429689 13293 raft_consensus.cc:686] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [term 1 LEADER]: Becoming Leader. State: Replica: 62dd20faf1fa4f76b20695746b7a2bcc, State: Running, Role: LEADER
I20260810 14:44:36.430050 13293 consensus_queue.cc:237] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d602ebadc4044b16e3b9b67876d6e" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 42133 } } peers { permanent_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 35129 } } peers { permanent_uuid: "24e371f6b4724a82a448e05f5dbbd23b" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 36013 } }
W20260810 14:44:36.431882 13259 tablet.cc:2369] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:44:36.433751 12548 catalog_manager.cc:5495] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc reported cstate change: term changed from 0 to 1, leader changed from <none> to 62dd20faf1fa4f76b20695746b7a2bcc (127.15.84.1). New cstate: current_term: 1 leader_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d602ebadc4044b16e3b9b67876d6e" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 42133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62dd20faf1fa4f76b20695746b7a2bcc" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 35129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24e371f6b4724a82a448e05f5dbbd23b" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 36013 } health_report { overall_health: UNKNOWN } } }
I20260810 14:44:36.463910 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.460149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38288 (local address 127.15.84.1:35129)
0810 14:44:36.460293 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:36.460297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:36.460313 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:44:36.460421 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:36.460423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:36.460485 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:36.460594 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:44:36.460887 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.461474 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.462328 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:36.462549 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:36.462642 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:36.462984 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:44:36.463428 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:44:36.463574 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:36.463699 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 14:44:36.463740 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:44:36.469960 12984 raft_consensus.cc:1261] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e [term 1 FOLLOWER]: Refusing update from remote peer 62dd20faf1fa4f76b20695746b7a2bcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:44:36.469961 13189 raft_consensus.cc:1261] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b [term 1 FOLLOWER]: Refusing update from remote peer 62dd20faf1fa4f76b20695746b7a2bcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:44:36.470677 13293 consensus_queue.cc:1035] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a61d602ebadc4044b16e3b9b67876d6e" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:44:36.470870 13300 consensus_queue.cc:1035] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "24e371f6b4724a82a448e05f5dbbd23b" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:44:36.478684 13305 mvcc.cc:205] Tried to move back new op lower bound from 7316984121214025728 to 7316984121060614144. Current Snapshot: MvccSnapshot[applied={T|T < 7316984121214025728}]
I20260810 14:44:36.487897 12547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:44:36.488078 12547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:44:36.490563 12547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9440a9f512cc4fc69da1dd2787b527ed: Sending DeleteTablet for 3 replicas of tablet c969a805890441969e3681de4b0f2e3d
I20260810 14:44:36.491325 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet c969a805890441969e3681de4b0f2e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:36 UTC) from {username='slave'} at 127.0.0.1:56764
I20260810 14:44:36.491437 13159 tablet_service.cc:1514] Processing DeleteTablet for tablet c969a805890441969e3681de4b0f2e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:36 UTC) from {username='slave'} at 127.0.0.1:38146
I20260810 14:44:36.491534 12741 tablet_service.cc:1514] Processing DeleteTablet for tablet c969a805890441969e3681de4b0f2e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:36 UTC) from {username='slave'} at 127.0.0.1:38272
I20260810 14:44:36.491840 13324 tablet_replica.cc:330] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e: stopping tablet replica
I20260810 14:44:36.491919 13325 tablet_replica.cc:330] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b: stopping tablet replica
I20260810 14:44:36.492163 13324 raft_consensus.cc:2229] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:44:36.492214 13325 raft_consensus.cc:2229] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:44:36.492501 13325 raft_consensus.cc:2258] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:44:36.492499 13324 raft_consensus.cc:2258] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:44:36.492959 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 12643
I20260810 14:44:36.492992 13326 tablet_replica.cc:330] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc: stopping tablet replica
I20260810 14:44:36.493270 13326 raft_consensus.cc:2229] T c969a805890441969e3681de4b0f2e3d P 62dd20faf1fa4f76b20695746b7a2bcc [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:44:36.494079 13324 ts_tablet_manager.cc:1905] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:44:36.494079 13325 ts_tablet_manager.cc:1905] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:44:36.496873 13325 ts_tablet_manager.cc:1918] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:44:36.496873 13324 ts_tablet_manager.cc:1918] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:44:36.496984 13325 log.cc:1196] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/c969a805890441969e3681de4b0f2e3d
I20260810 14:44:36.496984 13324 log.cc:1196] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/c969a805890441969e3681de4b0f2e3d
I20260810 14:44:36.497342 13324 ts_tablet_manager.cc:1939] T c969a805890441969e3681de4b0f2e3d P a61d602ebadc4044b16e3b9b67876d6e: Deleting consensus metadata
I20260810 14:44:36.497342 13325 ts_tablet_manager.cc:1939] T c969a805890441969e3681de4b0f2e3d P 24e371f6b4724a82a448e05f5dbbd23b: Deleting consensus metadata
I20260810 14:44:36.498769 12526 catalog_manager.cc:4841] TS 24e371f6b4724a82a448e05f5dbbd23b (127.15.84.3:36013): tablet c969a805890441969e3681de4b0f2e3d (table test-table [id=4e6ccae09040401ba56e5a200a806fc1]) successfully deleted
I20260810 14:44:36.498867 12527 catalog_manager.cc:4841] TS a61d602ebadc4044b16e3b9b67876d6e (127.15.84.2:42133): tablet c969a805890441969e3681de4b0f2e3d (table test-table [id=4e6ccae09040401ba56e5a200a806fc1]) successfully deleted
W20260810 14:44:36.504923 12532 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:35129 (error 108)
I20260810 14:44:36.505012 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 12856
I20260810 14:44:36.510309 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:36.509949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:35129 (local address 127.0.0.1:38302)
0810 14:44:36.510141 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:36.510212 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:35129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260810 14:44:36.510493 12532 catalog_manager.cc:4584] TS 62dd20faf1fa4f76b20695746b7a2bcc (127.15.84.1:35129): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c969a805890441969e3681de4b0f2e3d: Network error: Client connection negotiation failed: client connection to 127.15.84.1:35129: connect: Connection refused (error 111)
I20260810 14:44:36.519001 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 13058
I20260810 14:44:36.535733 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 12502
2026-08-10T14:44:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1235 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:36 dist-test-slave-rmhg krb5kdc[13333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:36 dist-test-slave-rmhg krb5kdc[13333](info): set up 2 sockets
Aug 10 14:44:36 dist-test-slave-rmhg krb5kdc[13333](info): commencing operation
krb5kdc: starting...
W20260810 14:44:37.704064 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:37 dist-test-slave-rmhg krb5kdc[13333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:44:37.843983 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:34603
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:34603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:37.959846 13354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:37.960139 13354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:37.960194 13354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:37.964306 13354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:37.964394 13354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:37.964423 13354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:37.964444 13354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:37.964483 13354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:37.970059 13354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:34603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:34603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:37.971262 13354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:37.972335 13354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:37.981050 13360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:37.981256 13354 server_base.cc:1034] running on GCE node
W20260810 14:44:37.981065 13364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:37.981050 13359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:37.981976 13354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:37.983137 13354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:37.984310 13354 hybrid_clock.cc:648] HybridClock initialized: now 1786373077984267 us; error 62 us; skew 500 ppm
Aug 10 14:44:37 dist-test-slave-rmhg krb5kdc[13333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373077, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:37.987092 13354 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:44:37.987979 13354 webserver.cc:460] Webserver started at http://127.15.84.62:33585/ using document root <none> and password file <none>
I20260810 14:44:37.988518 13354 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:37.988577 13354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:37.988759 13354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:37.990829 13354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "7eec7885f7be41af9abe3d6f20457476"
format_stamp: "Formatted at 2026-08-10 14:44:37 on dist-test-slave-rmhg"
server_key: "c91deef73d32602f05eb6e609d25e24b"
server_key_iv: "62207159187d26518b83da010761c8c1"
server_key_version: "encryptionkey@0"
I20260810 14:44:37.991354 13354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "7eec7885f7be41af9abe3d6f20457476"
format_stamp: "Formatted at 2026-08-10 14:44:37 on dist-test-slave-rmhg"
server_key: "c91deef73d32602f05eb6e609d25e24b"
server_key_iv: "62207159187d26518b83da010761c8c1"
server_key_version: "encryptionkey@0"
I20260810 14:44:37.995222 13354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:44:37.997731 13372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:37.998989 13354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:44:37.999116 13354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "7eec7885f7be41af9abe3d6f20457476"
format_stamp: "Formatted at 2026-08-10 14:44:37 on dist-test-slave-rmhg"
server_key: "c91deef73d32602f05eb6e609d25e24b"
server_key_iv: "62207159187d26518b83da010761c8c1"
server_key_version: "encryptionkey@0"
I20260810 14:44:37.999222 13354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:38.017105 13354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:38.020596 13354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:38.020776 13354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:38.027995 13354 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:34603
I20260810 14:44:38.028102 13447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:34603 every 8 connection(s)
I20260810 14:44:38.029037 13354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:38.031432 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 13354
I20260810 14:44:38.031596 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:38.031877 15696 external_mini_cluster.cc:1428] Setting key e337c4dd17184a052fc1444ab70fc861
I20260810 14:44:38.032133 13448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:38.039705 13448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476: Bootstrap starting.
Aug 10 14:44:38 dist-test-slave-rmhg krb5kdc[13333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:44:38.042383 13448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:38.043380 13448 log.cc:824] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:38.045871 13448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476: No bootstrap required, opened a new log
I20260810 14:44:38.047335 13455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:38.033256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.15.84.62:34603)
0810 14:44:38.033699 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:38.033720 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:38.033767 (+ 47us) server_negotiation.cc:407] Connection header received
0810 14:44:38.034552 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:38.034579 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:38.035131 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:38.035539 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:38.036259 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.037778 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.038689 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.039407 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.039573 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:38.042275 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:38.042282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:38.042301 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:38.042340 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:38.044103 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:38.044558 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:38.044565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:38.044571 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:38.044711 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:38.044939 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:38.044942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:38.044944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:38.045522 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:38.045730 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:38.045930 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 14:44:38.046269 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":132,"threads_started":1}
I20260810 14:44:38.050268 13448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eec7885f7be41af9abe3d6f20457476" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34603 } }
I20260810 14:44:38.050511 13448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:38.050554 13448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eec7885f7be41af9abe3d6f20457476, State: Initialized, Role: FOLLOWER
I20260810 14:44:38.050940 13448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eec7885f7be41af9abe3d6f20457476" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34603 } }
I20260810 14:44:38.051059 13448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:38.051118 13448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:38.051216 13448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:38.052285 13448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eec7885f7be41af9abe3d6f20457476" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34603 } }
I20260810 14:44:38.052604 13448 leader_election.cc:302] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eec7885f7be41af9abe3d6f20457476; no voters:
I20260810 14:44:38.052912 13448 leader_election.cc:288] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:38.053079 13458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:38.053427 13458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [term 1 LEADER]: Becoming Leader. State: Replica: 7eec7885f7be41af9abe3d6f20457476, State: Running, Role: LEADER
I20260810 14:44:38.053768 13458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eec7885f7be41af9abe3d6f20457476" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34603 } }
I20260810 14:44:38.054139 13448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:38.055342 13462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eec7885f7be41af9abe3d6f20457476. Latest consensus state: current_term: 1 leader_uuid: "7eec7885f7be41af9abe3d6f20457476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eec7885f7be41af9abe3d6f20457476" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34603 } } }
I20260810 14:44:38.055459 13462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:38.055860 13472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:38.056107 13461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eec7885f7be41af9abe3d6f20457476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eec7885f7be41af9abe3d6f20457476" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 34603 } } }
I20260810 14:44:38.056265 13461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:38.059343 13472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:38.065523 13472 catalog_manager.cc:1347] Generated new cluster ID: 96aaf823bdad4e8caf8c7c3fda0dd8b1
I20260810 14:44:38.065606 13472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:38.076838 13472 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:38.078063 13472 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:38.087438 13472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476: Generated new TSK 0
I20260810 14:44:38.088209 13472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
Entry for principal kudu/127.15.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab.
I20260810 14:44:38.128993 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34603
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:38.252074 13492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:38.252363 13492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:38.252406 13492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:38.256744 13492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:38.256809 13492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:38.256879 13492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:44:38.262382 13492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.1.keytab
--principal=kudu/127.15.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:38.263499 13492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:38.264562 13492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:38.274523 13502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:38.274524 13500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:38.274572 13492 server_base.cc:1034] running on GCE node
W20260810 14:44:38.274523 13505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:38.275219 13492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:38.275918 13492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:38.277078 13492 hybrid_clock.cc:648] HybridClock initialized: now 1786373078277048 us; error 48 us; skew 500 ppm
Aug 10 14:44:38 dist-test-slave-rmhg krb5kdc[13333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373078, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:38.280041 13492 init.cc:377] Logged in from keytab as kudu/127.15.84.1@KRBTEST.COM (short username kudu)
I20260810 14:44:38.281127 13492 webserver.cc:460] Webserver started at http://127.15.84.1:35977/ using document root <none> and password file <none>
I20260810 14:44:38.281754 13492 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:38.281824 13492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:38.282044 13492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:38.284219 13492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "626a692bccb24eddb498d5dce99613c4"
format_stamp: "Formatted at 2026-08-10 14:44:38 on dist-test-slave-rmhg"
server_key: "12eea9e58664ded9d4ad99eb00b143e2"
server_key_iv: "3e2b0264d14d78ffecdc59d74798f583"
server_key_version: "encryptionkey@0"
I20260810 14:44:38.284817 13492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "626a692bccb24eddb498d5dce99613c4"
format_stamp: "Formatted at 2026-08-10 14:44:38 on dist-test-slave-rmhg"
server_key: "12eea9e58664ded9d4ad99eb00b143e2"
server_key_iv: "3e2b0264d14d78ffecdc59d74798f583"
server_key_version: "encryptionkey@0"
I20260810 14:44:38.288985 13492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 14:44:38.291760 13512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:38.293210 13492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:44:38.293354 13492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "626a692bccb24eddb498d5dce99613c4"
format_stamp: "Formatted at 2026-08-10 14:44:38 on dist-test-slave-rmhg"
server_key: "12eea9e58664ded9d4ad99eb00b143e2"
server_key_iv: "3e2b0264d14d78ffecdc59d74798f583"
server_key_version: "encryptionkey@0"
I20260810 14:44:38.293476 13492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:38.321305 13492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:38.324957 13492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:38.325223 13492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:38.325903 13492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:38.326830 13492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:38.326881 13492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:38.326939 13492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:38.326967 13492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:38.336125 13492 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:40973
I20260810 14:44:38.336207 13692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:40973 every 8 connection(s)
I20260810 14:44:38.337138 13492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:44:38.337934 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 13492
I20260810 14:44:38.338089 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:44:38.338403 15696 external_mini_cluster.cc:1428] Setting key 38c483cfac4ef4f3fe87b3c12a9b69c8
Aug 10 14:44:38 dist-test-slave-rmhg krb5kdc[13333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373078, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.1@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.2@KRBTEST.COM" created.
I20260810 14:44:38.353269 13455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:38.339296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:57127 (local address 127.15.84.62:34603)
0810 14:44:38.339426 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:38.339429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:38.340642 (+ 1213us) server_negotiation.cc:407] Connection header received
0810 14:44:38.342053 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:38.342058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:38.342120 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:38.342212 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:38.343875 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.344702 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.346028 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.346382 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.346505 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:38.349404 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:38.349409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:38.349412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:38.349445 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:38.351124 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:38.351637 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:38.351640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:38.351641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:38.351724 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:38.352019 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:38.352022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:38.352023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:38.352235 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:38.352312 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:38.352948 (+ 636us) server_negotiation.cc:299] Negotiation successful
0810 14:44:38.353074 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 14:44:38.354257 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:38.339884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:34603 (local address 127.15.84.1:57127)
0810 14:44:38.340411 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:38.340472 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:38.341817 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:38.342435 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:38.342446 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:38.342962 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:38.343698 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:38.343717 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.344830 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.344833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:38.345887 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:38.345899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.346537 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.346540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:38.346697 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:38.347342 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:44:38.347360 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:38.349183 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:38.351239 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:38.351246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:38.351261 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:38.351529 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:38.351796 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:38.351798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:38.351799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:38.351936 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:38.352307 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:38.352312 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:38.352627 (+ 315us) client_negotiation.cc:769] Sending connection context
0810 14:44:38.352883 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 14:44:38.353294 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":138,"threads_started":1}
I20260810 14:44:38.355778 13693 heartbeater.cc:347] Connected to a master server at 127.15.84.62:34603
I20260810 14:44:38.356055 13693 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:38.356570 13693 heartbeater.cc:511] Master 127.15.84.62:34603 requested a full tablet report, sending...
I20260810 14:44:38.358253 13394 ts_manager.cc:194] Registered new tserver with Master: 626a692bccb24eddb498d5dce99613c4 (127.15.84.1:40973)
I20260810 14:44:38.359436 13394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.1@KRBTEST.COM'} at 127.15.84.1:57127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
Entry for principal kudu/127.15.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab.
I20260810 14:44:38.371603 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34603
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:38.490371 13704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:38.490670 13704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:38.490736 13704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:38.494899 13704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:38.494980 13704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:38.495072 13704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:44:38.500757 13704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.2.keytab
--principal=kudu/127.15.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:38.501987 13704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:38.503227 13704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:38.513677 13712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:38.513686 13714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:38.513748 13704 server_base.cc:1034] running on GCE node
W20260810 14:44:38.513679 13711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:38.514359 13704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:38.515100 13704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:38.516279 13704 hybrid_clock.cc:648] HybridClock initialized: now 1786373078516244 us; error 47 us; skew 500 ppm
Aug 10 14:44:38 dist-test-slave-rmhg krb5kdc[13333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373078, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:38.519356 13704 init.cc:377] Logged in from keytab as kudu/127.15.84.2@KRBTEST.COM (short username kudu)
I20260810 14:44:38.520367 13704 webserver.cc:460] Webserver started at http://127.15.84.2:33193/ using document root <none> and password file <none>
I20260810 14:44:38.520961 13704 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:38.521036 13704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:38.521306 13704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:38.523625 13704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "79c49ea313ec48b09e14e6d741217964"
format_stamp: "Formatted at 2026-08-10 14:44:38 on dist-test-slave-rmhg"
server_key: "30816ec77feb780677e258adced6aa3f"
server_key_iv: "3636f176e4d775ca33070c28f4ebb02b"
server_key_version: "encryptionkey@0"
I20260810 14:44:38.524197 13704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "79c49ea313ec48b09e14e6d741217964"
format_stamp: "Formatted at 2026-08-10 14:44:38 on dist-test-slave-rmhg"
server_key: "30816ec77feb780677e258adced6aa3f"
server_key_iv: "3636f176e4d775ca33070c28f4ebb02b"
server_key_version: "encryptionkey@0"
I20260810 14:44:38.528393 13704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:44:38.531405 13724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:38.532838 13704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 14:44:38.532986 13704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "79c49ea313ec48b09e14e6d741217964"
format_stamp: "Formatted at 2026-08-10 14:44:38 on dist-test-slave-rmhg"
server_key: "30816ec77feb780677e258adced6aa3f"
server_key_iv: "3636f176e4d775ca33070c28f4ebb02b"
server_key_version: "encryptionkey@0"
I20260810 14:44:38.533136 13704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:38.545589 13704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:38.548905 13704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:38.549139 13704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:38.549836 13704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:38.550765 13704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:38.550815 13704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:38.550873 13704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:38.550902 13704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:38.560464 13704 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:36969
I20260810 14:44:38.560500 13890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:36969 every 8 connection(s)
I20260810 14:44:38.561477 13704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:44:38.569685 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 13704
I20260810 14:44:38.569837 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:44:38.570127 15696 external_mini_cluster.cc:1428] Setting key 1aab44ed55c1522c5dc87287e4fc8015
Aug 10 14:44:38 dist-test-slave-rmhg krb5kdc[13333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373078, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.2@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:44:38.576427 13455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:38.563447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:36253 (local address 127.15.84.62:34603)
0810 14:44:38.563648 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:38.563652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:38.564601 (+ 949us) server_negotiation.cc:407] Connection header received
0810 14:44:38.566054 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:38.566059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:38.566125 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:38.566204 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:38.567534 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.568084 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.569416 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.569725 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.569836 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:38.572681 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:38.572685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:38.572688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:38.572720 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:38.574213 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:38.574788 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:38.574792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:38.574794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:38.574882 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:38.575235 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:38.575238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:38.575240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:38.575444 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:38.575533 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:38.576112 (+ 579us) server_negotiation.cc:299] Negotiation successful
0810 14:44:38.576258 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 14:44:38.577363 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:38.563890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:34603 (local address 127.15.84.2:36253)
0810 14:44:38.564407 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:38.564455 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:38.565803 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:38.566414 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:38.566422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:38.566840 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:38.567376 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:38.567392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.568230 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.568234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:38.569265 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:38.569277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.569947 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.569951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:38.570131 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:38.570787 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:44:38.570806 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:38.572490 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:38.574346 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:38.574354 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:38.574373 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:38.574683 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:38.574974 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:38.574977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:38.574979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:38.575143 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:38.575556 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:38.575564 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:38.575855 (+ 291us) client_negotiation.cc:769] Sending connection context
0810 14:44:38.576044 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 14:44:38.576422 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":133,"threads_started":1}
I20260810 14:44:38.578629 13892 heartbeater.cc:347] Connected to a master server at 127.15.84.62:34603
I20260810 14:44:38.578883 13892 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:38.579411 13892 heartbeater.cc:511] Master 127.15.84.62:34603 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.84.3@KRBTEST.COM; defaulting to no policy
I20260810 14:44:38.580540 13394 ts_manager.cc:194] Registered new tserver with Master: 79c49ea313ec48b09e14e6d741217964 (127.15.84.2:36969)
I20260810 14:44:38.581139 13394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.2@KRBTEST.COM'} at 127.15.84.2:36253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
Entry for principal kudu/127.15.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab.
I20260810 14:44:38.600145 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34603
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:38.721498 13901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:38.721805 13901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:38.721853 13901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:38.726099 13901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:38.726192 13901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:38.726256 13901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:44:38.732474 13901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.3.keytab
--principal=kudu/127.15.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:34603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:38.733700 13901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:38.734752 13901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:38.745030 13913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:38.745042 13909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:38.745042 13907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:38.745416 13901 server_base.cc:1034] running on GCE node
I20260810 14:44:38.745890 13901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:38.746579 13901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:38.747771 13901 hybrid_clock.cc:648] HybridClock initialized: now 1786373078747746 us; error 38 us; skew 500 ppm
Aug 10 14:44:38 dist-test-slave-rmhg krb5kdc[13333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373078, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:38.750684 13901 init.cc:377] Logged in from keytab as kudu/127.15.84.3@KRBTEST.COM (short username kudu)
I20260810 14:44:38.751649 13901 webserver.cc:460] Webserver started at http://127.15.84.3:40549/ using document root <none> and password file <none>
I20260810 14:44:38.752218 13901 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:38.752290 13901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:38.752491 13901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:38.754822 13901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "efbf8ba136af47758ec2f66e935dccd1"
format_stamp: "Formatted at 2026-08-10 14:44:38 on dist-test-slave-rmhg"
server_key: "119ad0ed7e23509ac9e5ce3ec83224e5"
server_key_iv: "7f84d643a2a9af031aea2f9107421436"
server_key_version: "encryptionkey@0"
I20260810 14:44:38.755389 13901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "efbf8ba136af47758ec2f66e935dccd1"
format_stamp: "Formatted at 2026-08-10 14:44:38 on dist-test-slave-rmhg"
server_key: "119ad0ed7e23509ac9e5ce3ec83224e5"
server_key_iv: "7f84d643a2a9af031aea2f9107421436"
server_key_version: "encryptionkey@0"
I20260810 14:44:38.759316 13901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:44:38.761929 13926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:38.763366 13901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:44:38.763504 13901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "efbf8ba136af47758ec2f66e935dccd1"
format_stamp: "Formatted at 2026-08-10 14:44:38 on dist-test-slave-rmhg"
server_key: "119ad0ed7e23509ac9e5ce3ec83224e5"
server_key_iv: "7f84d643a2a9af031aea2f9107421436"
server_key_version: "encryptionkey@0"
I20260810 14:44:38.763610 13901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:38.773635 13901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:38.776846 13901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:38.777037 13901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:38.777693 13901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:38.778579 13901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:38.778627 13901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:38.778681 13901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:38.778708 13901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:38.787374 13901 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:46201
I20260810 14:44:38.787397 14104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:46201 every 8 connection(s)
I20260810 14:44:38.788304 13901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:44:38.788683 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 13901
I20260810 14:44:38.788847 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:44:38.789222 15696 external_mini_cluster.cc:1428] Setting key 3bb0fac754097ab0e3cfe414e2180ecf
Aug 10 14:44:38 dist-test-slave-rmhg krb5kdc[13333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373078, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.3@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:44:38.802536 13455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:38.790370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:40091 (local address 127.15.84.62:34603)
0810 14:44:38.790518 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:38.790522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:38.791684 (+ 1162us) server_negotiation.cc:407] Connection header received
0810 14:44:38.793002 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:38.793005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:38.793063 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:38.793136 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:38.794492 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.795040 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.796230 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.796456 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.796546 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:38.799161 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:38.799165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:38.799167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:38.799193 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:38.800383 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:38.800887 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:38.800891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:38.800893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:38.800975 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:38.801367 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:38.801370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:38.801372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:38.801569 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:38.801669 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:38.802203 (+ 534us) server_negotiation.cc:299] Negotiation successful
0810 14:44:38.802340 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:44:38.803540 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:38.790953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:34603 (local address 127.15.84.3:40091)
0810 14:44:38.791521 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:38.791582 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:38.792792 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:38.793367 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:38.793375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:38.793797 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:38.794337 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:38.794353 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.795184 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.795187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:38.796100 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:38.796110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.796605 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.796609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:38.796786 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:38.797404 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:44:38.797422 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:38.798984 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:38.800494 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:38.800501 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:38.800518 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:38.800787 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:38.801060 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:44:38.801063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:44:38.801065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:44:38.801277 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:44:38.801673 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:38.801681 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:38.801957 (+ 276us) client_negotiation.cc:769] Sending connection context
0810 14:44:38.802124 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 14:44:38.802487 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":146,"threads_started":1}
I20260810 14:44:38.804816 14105 heartbeater.cc:347] Connected to a master server at 127.15.84.62:34603
I20260810 14:44:38.805083 14105 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:38.805692 14105 heartbeater.cc:511] Master 127.15.84.62:34603 requested a full tablet report, sending...
I20260810 14:44:38.806833 13394 ts_manager.cc:194] Registered new tserver with Master: efbf8ba136af47758ec2f66e935dccd1 (127.15.84.3:46201)
I20260810 14:44:38.807458 13394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.84.3@KRBTEST.COM'} at 127.15.84.3:40091
I20260810 14:44:38.816100 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:44:38.825546 13455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:38.818315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.15.84.62:34603)
0810 14:44:38.818460 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:38.818464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:38.818575 (+ 111us) server_negotiation.cc:407] Connection header received
0810 14:44:38.818685 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:38.818688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:38.818730 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:38.818801 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:38.819511 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.820047 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.820980 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.821260 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.821345 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:38.822661 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:38.822664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:38.822666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:38.822693 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:38.823943 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:38.824362 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:38.824364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:38.824366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:38.824447 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:38.824669 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:38.824671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:38.824673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:38.824898 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:38.825000 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:38.825283 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 14:44:38.825410 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 14:44:38.828759 13394 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:44:38.831928 13394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:44:38.847456 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:38.840921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:46201 (local address 127.0.0.1:34208)
0810 14:44:38.841700 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:38.841752 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:38.841886 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:38.842806 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:38.842812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:38.842837 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:38.842996 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:38.843005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.844557 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.844561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:38.845880 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:38.845890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.846962 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.846965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:38.847054 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:38.847088 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 14:44:38.847168 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 14:44:38.847293 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":27,"thread_start_us":98,"threads_started":1}
I20260810 14:44:38.847487 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:38.841096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34208 (local address 127.15.84.3:46201)
0810 14:44:38.842300 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:38.842312 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:38.842336 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:44:38.842419 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:38.842425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:38.842639 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:38.842780 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:38.843138 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.844400 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.846013 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.846830 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.846955 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:38.847136 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:38.847207 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 14:44:38.847316 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":202,"threads_started":1}
I20260810 14:44:38.847795 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:38.841853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.15.84.2:36969)
0810 14:44:38.842600 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:38.842612 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:38.842635 (+ 23us) server_negotiation.cc:407] Connection header received
0810 14:44:38.842700 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:38.842704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:38.842886 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:38.843044 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:38.843275 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.844400 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.846291 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.847128 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.847237 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:38.847378 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:38.847434 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 14:44:38.847526 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":118,"threads_started":1}
I20260810 14:44:38.847709 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:38.841321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:36969 (local address 127.0.0.1:58016)
0810 14:44:38.842249 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:38.842261 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:38.842335 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:38.843037 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:38.843040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:38.843058 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:38.843165 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:38.843172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.844605 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.844608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:38.846157 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:38.846175 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.847253 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.847257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:38.847333 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:38.847355 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:44:38.847409 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 14:44:38.847469 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":213,"thread_start_us":104,"threads_started":1}
I20260810 14:44:38.849323 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:38.841053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:40973 (local address 127.0.0.1:60790)
0810 14:44:38.841944 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:38.841958 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:38.842044 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:38.843476 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:38.843478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:38.843496 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:38.843587 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:38.843594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.844605 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.844608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:38.846157 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:38.846168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.848919 (+ 2751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.848923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:38.849022 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:38.849047 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:44:38.849096 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 14:44:38.849195 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":149,"thread_start_us":145,"threads_started":1}
I20260810 14:44:38.850028 14143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:38.842133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60790 (local address 127.15.84.1:40973)
0810 14:44:38.843072 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:38.843088 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:38.843110 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:44:38.843185 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:38.843190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:38.843358 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:38.843495 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:38.843686 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.844417 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.847961 (+ 3544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:38.848773 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:38.849580 (+ 807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:38.849738 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:38.849792 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 14:44:38.849884 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":124,"threads_started":1}
I20260810 14:44:38.851100 13798 tablet_service.cc:1467] Processing CreateTablet for tablet b4181504be474c42ac32814e884e8b6d (DEFAULT_TABLE table=test-table [id=5c0f97c5f9b0423e842f5efb1c25ddc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:38.851068 14003 tablet_service.cc:1467] Processing CreateTablet for tablet b4181504be474c42ac32814e884e8b6d (DEFAULT_TABLE table=test-table [id=5c0f97c5f9b0423e842f5efb1c25ddc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:38.852149 13798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4181504be474c42ac32814e884e8b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:38.852205 14003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4181504be474c42ac32814e884e8b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:38.852933 13590 tablet_service.cc:1467] Processing CreateTablet for tablet b4181504be474c42ac32814e884e8b6d (DEFAULT_TABLE table=test-table [id=5c0f97c5f9b0423e842f5efb1c25ddc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:38.854238 13590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4181504be474c42ac32814e884e8b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:38.859165 14144 tablet_bootstrap.cc:492] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964: Bootstrap starting.
I20260810 14:44:38.860714 14146 tablet_bootstrap.cc:492] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1: Bootstrap starting.
I20260810 14:44:38.861799 14144 tablet_bootstrap.cc:654] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:38.862600 14144 log.cc:824] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:38.862764 14149 tablet_bootstrap.cc:492] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4: Bootstrap starting.
I20260810 14:44:38.863031 14146 tablet_bootstrap.cc:654] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:38.864307 14146 log.cc:824] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:38.865027 14149 tablet_bootstrap.cc:654] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:38.865281 14144 tablet_bootstrap.cc:492] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964: No bootstrap required, opened a new log
I20260810 14:44:38.865492 14144 ts_tablet_manager.cc:1397] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 14:44:38.865796 14149 log.cc:824] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:38.866459 14146 tablet_bootstrap.cc:492] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1: No bootstrap required, opened a new log
I20260810 14:44:38.866652 14146 ts_tablet_manager.cc:1397] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:44:38.867596 14149 tablet_bootstrap.cc:492] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4: No bootstrap required, opened a new log
I20260810 14:44:38.867776 14149 ts_tablet_manager.cc:1397] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260810 14:44:38.869521 14144 raft_consensus.cc:348] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbf8ba136af47758ec2f66e935dccd1" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46201 } } peers { permanent_uuid: "626a692bccb24eddb498d5dce99613c4" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40973 } } peers { permanent_uuid: "79c49ea313ec48b09e14e6d741217964" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36969 } }
I20260810 14:44:38.869835 14144 raft_consensus.cc:374] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:38.869896 14144 raft_consensus.cc:729] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c49ea313ec48b09e14e6d741217964, State: Initialized, Role: FOLLOWER
I20260810 14:44:38.870376 14144 consensus_queue.cc:260] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbf8ba136af47758ec2f66e935dccd1" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46201 } } peers { permanent_uuid: "626a692bccb24eddb498d5dce99613c4" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40973 } } peers { permanent_uuid: "79c49ea313ec48b09e14e6d741217964" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36969 } }
I20260810 14:44:38.870808 14146 raft_consensus.cc:348] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbf8ba136af47758ec2f66e935dccd1" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46201 } } peers { permanent_uuid: "626a692bccb24eddb498d5dce99613c4" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40973 } } peers { permanent_uuid: "79c49ea313ec48b09e14e6d741217964" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36969 } }
I20260810 14:44:38.871187 14146 raft_consensus.cc:374] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:38.871256 14146 raft_consensus.cc:729] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbf8ba136af47758ec2f66e935dccd1, State: Initialized, Role: FOLLOWER
I20260810 14:44:38.871459 14144 ts_tablet_manager.cc:1428] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 14:44:38.871701 14146 consensus_queue.cc:260] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbf8ba136af47758ec2f66e935dccd1" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46201 } } peers { permanent_uuid: "626a692bccb24eddb498d5dce99613c4" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40973 } } peers { permanent_uuid: "79c49ea313ec48b09e14e6d741217964" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36969 } }
I20260810 14:44:38.871865 13892 heartbeater.cc:503] Master 127.15.84.62:34603 was elected leader, sending a full tablet report...
I20260810 14:44:38.871865 14149 raft_consensus.cc:348] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbf8ba136af47758ec2f66e935dccd1" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46201 } } peers { permanent_uuid: "626a692bccb24eddb498d5dce99613c4" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40973 } } peers { permanent_uuid: "79c49ea313ec48b09e14e6d741217964" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36969 } }
I20260810 14:44:38.872144 14149 raft_consensus.cc:374] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:38.872210 14149 raft_consensus.cc:729] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626a692bccb24eddb498d5dce99613c4, State: Initialized, Role: FOLLOWER
I20260810 14:44:38.872462 14105 heartbeater.cc:503] Master 127.15.84.62:34603 was elected leader, sending a full tablet report...
I20260810 14:44:38.872628 14149 consensus_queue.cc:260] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbf8ba136af47758ec2f66e935dccd1" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46201 } } peers { permanent_uuid: "626a692bccb24eddb498d5dce99613c4" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40973 } } peers { permanent_uuid: "79c49ea313ec48b09e14e6d741217964" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36969 } }
I20260810 14:44:38.873409 13693 heartbeater.cc:503] Master 127.15.84.62:34603 was elected leader, sending a full tablet report...
I20260810 14:44:38.873652 14149 ts_tablet_manager.cc:1428] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260810 14:44:38.874603 14146 ts_tablet_manager.cc:1428] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260810 14:44:39.039645 14106 tablet.cc:2369] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:44:39.062757 13893 tablet.cc:2369] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:44:39.065380 14158 raft_consensus.cc:482] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:44:39.065629 14158 raft_consensus.cc:504] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbf8ba136af47758ec2f66e935dccd1" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46201 } } peers { permanent_uuid: "626a692bccb24eddb498d5dce99613c4" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40973 } } peers { permanent_uuid: "79c49ea313ec48b09e14e6d741217964" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36969 } }
I20260810 14:44:39.066929 14158 leader_election.cc:288] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efbf8ba136af47758ec2f66e935dccd1 (127.15.84.3:46201), 79c49ea313ec48b09e14e6d741217964 (127.15.84.2:36969)
I20260810 14:44:39.070956 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:39.067529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:55445 (local address 127.15.84.3:46201)
0810 14:44:39.067669 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:39.067674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:39.067689 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:44:39.067745 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:39.067748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:39.067794 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:39.067859 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:39.068146 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.068692 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.069964 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.070395 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.070489 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:39.070562 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:39.070757 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 14:44:39.070829 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:44:39.070968 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:39.067253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:55373 (local address 127.15.84.2:36969)
0810 14:44:39.067406 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:39.067410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:39.067732 (+ 322us) server_negotiation.cc:407] Connection header received
0810 14:44:39.067853 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:39.067856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:39.067899 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:39.067979 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:39.068244 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.068788 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.069974 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.070368 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.070462 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:39.070533 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:39.070781 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 14:44:39.070858 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:44:39.070988 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:39.067155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:36969 (local address 127.15.84.1:55373)
0810 14:44:39.067635 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:39.067646 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:39.067752 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:39.067987 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:39.067989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:39.068005 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:39.068138 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:39.068146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.068890 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.068894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:39.069845 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:39.069855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.070489 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.070493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:39.070643 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:39.070680 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 14:44:39.070745 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 14:44:39.070819 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":139,"thread_start_us":114,"threads_started":1}
I20260810 14:44:39.070982 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:39.067155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:46201 (local address 127.15.84.1:55445)
0810 14:44:39.067511 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:39.067526 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:39.067641 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:39.067879 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:39.067882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:39.067899 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:39.068050 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:39.068059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.068804 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.068807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:39.069843 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:39.069854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.070505 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.070508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:39.070632 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:39.070667 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 14:44:39.070731 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 14:44:39.070810 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":61,"thread_start_us":151,"threads_started":1}
I20260810 14:44:39.071756 13824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4181504be474c42ac32814e884e8b6d" candidate_uuid: "626a692bccb24eddb498d5dce99613c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79c49ea313ec48b09e14e6d741217964" is_pre_election: true
I20260810 14:44:39.071965 14033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4181504be474c42ac32814e884e8b6d" candidate_uuid: "626a692bccb24eddb498d5dce99613c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbf8ba136af47758ec2f66e935dccd1" is_pre_election: true
I20260810 14:44:39.072093 13824 raft_consensus.cc:2454] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 626a692bccb24eddb498d5dce99613c4 in term 0.
I20260810 14:44:39.072247 14033 raft_consensus.cc:2454] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 626a692bccb24eddb498d5dce99613c4 in term 0.
I20260810 14:44:39.072613 13517 leader_election.cc:302] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 626a692bccb24eddb498d5dce99613c4, 79c49ea313ec48b09e14e6d741217964, efbf8ba136af47758ec2f66e935dccd1; no voters:
I20260810 14:44:39.072921 14158 raft_consensus.cc:2781] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:44:39.073007 14158 raft_consensus.cc:482] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:44:39.073042 14158 raft_consensus.cc:3037] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:39.074213 14158 raft_consensus.cc:504] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbf8ba136af47758ec2f66e935dccd1" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46201 } } peers { permanent_uuid: "626a692bccb24eddb498d5dce99613c4" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40973 } } peers { permanent_uuid: "79c49ea313ec48b09e14e6d741217964" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36969 } }
I20260810 14:44:39.074582 14158 leader_election.cc:288] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [CANDIDATE]: Term 1 election: Requested vote from peers efbf8ba136af47758ec2f66e935dccd1 (127.15.84.3:46201), 79c49ea313ec48b09e14e6d741217964 (127.15.84.2:36969)
I20260810 14:44:39.074954 14033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4181504be474c42ac32814e884e8b6d" candidate_uuid: "626a692bccb24eddb498d5dce99613c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbf8ba136af47758ec2f66e935dccd1"
I20260810 14:44:39.075104 14033 raft_consensus.cc:3037] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:39.076269 14033 raft_consensus.cc:2454] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 626a692bccb24eddb498d5dce99613c4 in term 1.
I20260810 14:44:39.076546 13824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4181504be474c42ac32814e884e8b6d" candidate_uuid: "626a692bccb24eddb498d5dce99613c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79c49ea313ec48b09e14e6d741217964"
I20260810 14:44:39.076594 13516 leader_election.cc:302] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 626a692bccb24eddb498d5dce99613c4, efbf8ba136af47758ec2f66e935dccd1; no voters:
I20260810 14:44:39.076685 13824 raft_consensus.cc:3037] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:39.076766 14158 raft_consensus.cc:2781] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:39.076972 14158 raft_consensus.cc:686] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [term 1 LEADER]: Becoming Leader. State: Replica: 626a692bccb24eddb498d5dce99613c4, State: Running, Role: LEADER
I20260810 14:44:39.077366 14158 consensus_queue.cc:237] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbf8ba136af47758ec2f66e935dccd1" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46201 } } peers { permanent_uuid: "626a692bccb24eddb498d5dce99613c4" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40973 } } peers { permanent_uuid: "79c49ea313ec48b09e14e6d741217964" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36969 } }
I20260810 14:44:39.077885 13824 raft_consensus.cc:2454] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 626a692bccb24eddb498d5dce99613c4 in term 1.
I20260810 14:44:39.080736 13394 catalog_manager.cc:5495] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 626a692bccb24eddb498d5dce99613c4 (127.15.84.1). New cstate: current_term: 1 leader_uuid: "626a692bccb24eddb498d5dce99613c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbf8ba136af47758ec2f66e935dccd1" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "626a692bccb24eddb498d5dce99613c4" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 40973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79c49ea313ec48b09e14e6d741217964" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36969 } health_report { overall_health: UNKNOWN } } }
W20260810 14:44:39.088639 13699 tablet.cc:2369] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:44:39.141322 14143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:39.137629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60806 (local address 127.15.84.1:40973)
0810 14:44:39.137759 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:39.137764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:39.137779 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:44:39.137971 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:39.137975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:39.138039 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:39.138145 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:44:39.138410 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.138928 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.139801 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.140015 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.140113 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:39.140437 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:44:39.140887 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:44:39.141004 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:39.141075 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 14:44:39.141142 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:44:39.147670 13824 raft_consensus.cc:1261] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964 [term 1 FOLLOWER]: Refusing update from remote peer 626a692bccb24eddb498d5dce99613c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:44:39.147670 14033 raft_consensus.cc:1261] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1 [term 1 FOLLOWER]: Refusing update from remote peer 626a692bccb24eddb498d5dce99613c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:44:39.148598 14162 consensus_queue.cc:1035] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efbf8ba136af47758ec2f66e935dccd1" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 46201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:44:39.148805 14158 consensus_queue.cc:1035] T b4181504be474c42ac32814e884e8b6d P 626a692bccb24eddb498d5dce99613c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79c49ea313ec48b09e14e6d741217964" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 36969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:44:39.156332 14166 mvcc.cc:205] Tried to move back new op lower bound from 7316984132180910080 to 7316984131904143360. Current Snapshot: MvccSnapshot[applied={T|T < 7316984132180910080}]
I20260810 14:44:39.158206 14168 mvcc.cc:205] Tried to move back new op lower bound from 7316984132180910080 to 7316984131904143360. Current Snapshot: MvccSnapshot[applied={T|T < 7316984132180910080}]
I20260810 14:44:39.159754 14167 mvcc.cc:205] Tried to move back new op lower bound from 7316984132180910080 to 7316984131904143360. Current Snapshot: MvccSnapshot[applied={T|T < 7316984132180910080}]
I20260810 14:44:39.165518 13392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:44:39.165691 13392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:44:39.168211 13392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eec7885f7be41af9abe3d6f20457476: Sending DeleteTablet for 3 replicas of tablet b4181504be474c42ac32814e884e8b6d
I20260810 14:44:39.169023 14003 tablet_service.cc:1514] Processing DeleteTablet for tablet b4181504be474c42ac32814e884e8b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:39 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:34208
I20260810 14:44:39.169041 13590 tablet_service.cc:1514] Processing DeleteTablet for tablet b4181504be474c42ac32814e884e8b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:39 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:60790
I20260810 14:44:39.169132 13798 tablet_service.cc:1514] Processing DeleteTablet for tablet b4181504be474c42ac32814e884e8b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:39 UTC) from {username='kudu', principal='kudu/127.15.84.62@KRBTEST.COM'} at 127.0.0.1:58016
I20260810 14:44:39.169747 14178 tablet_replica.cc:330] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1: stopping tablet replica
I20260810 14:44:39.170114 14178 raft_consensus.cc:2229] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:44:39.170485 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 13492
I20260810 14:44:39.170500 14178 raft_consensus.cc:2258] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:44:39.171370 14180 tablet_replica.cc:330] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964: stopping tablet replica
I20260810 14:44:39.171674 14180 raft_consensus.cc:2229] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:44:39.171939 14180 raft_consensus.cc:2258] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:44:39.174988 14180 ts_tablet_manager.cc:1905] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:44:39.175017 14178 ts_tablet_manager.cc:1905] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:44:39.177523 14180 ts_tablet_manager.cc:1918] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:44:39.177623 14180 log.cc:1196] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/b4181504be474c42ac32814e884e8b6d
I20260810 14:44:39.177884 14180 ts_tablet_manager.cc:1939] T b4181504be474c42ac32814e884e8b6d P 79c49ea313ec48b09e14e6d741217964: Deleting consensus metadata
I20260810 14:44:39.178380 14178 ts_tablet_manager.cc:1918] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:44:39.178524 14178 log.cc:1196] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/b4181504be474c42ac32814e884e8b6d
I20260810 14:44:39.178884 14178 ts_tablet_manager.cc:1939] T b4181504be474c42ac32814e884e8b6d P efbf8ba136af47758ec2f66e935dccd1: Deleting consensus metadata
I20260810 14:44:39.179279 13377 catalog_manager.cc:4841] TS 79c49ea313ec48b09e14e6d741217964 (127.15.84.2:36969): tablet b4181504be474c42ac32814e884e8b6d (table test-table [id=5c0f97c5f9b0423e842f5efb1c25ddc3]) successfully deleted
I20260810 14:44:39.180131 13376 catalog_manager.cc:4841] TS efbf8ba136af47758ec2f66e935dccd1 (127.15.84.3:46201): tablet b4181504be474c42ac32814e884e8b6d (table test-table [id=5c0f97c5f9b0423e842f5efb1c25ddc3]) successfully deleted
W20260810 14:44:39.186072 13375 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:40973 (error 108)
I20260810 14:44:39.186172 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 13704
I20260810 14:44:39.202823 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:39.202029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:40973 (local address 127.0.0.1:60814)
0810 14:44:39.202628 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:39.202725 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474}
W20260810 14:44:39.203040 13375 catalog_manager.cc:4584] TS 626a692bccb24eddb498d5dce99613c4 (127.15.84.1:40973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4181504be474c42ac32814e884e8b6d: Network error: Client connection negotiation failed: client connection to 127.15.84.1:40973: connect: Connection refused (error 111)
I20260810 14:44:39.208534 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 13901
I20260810 14:44:39.220417 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 13354
2026-08-10T14:44:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2686 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T14:44:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:39Z Disabled control of system clock
I20260810 14:44:39.263171 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:33677
--webserver_interface=127.15.84.62
--webserver_port=0
--builtin_ntp_servers=127.15.84.20:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:33677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:44:39.381740 14187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:39.382030 14187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:39.382066 14187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:39.386210 14187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:39.386291 14187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:39.386304 14187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:39.386320 14187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:39.386335 14187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:39.391330 14187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:33677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:33677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:39.392583 14187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:39.394084 14187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:39.402606 14195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:39.402604 14194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:39.402854 14187 server_base.cc:1034] running on GCE node
W20260810 14:44:39.402609 14198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:39.403539 14187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:39.404734 14187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:39.405911 14187 hybrid_clock.cc:648] HybridClock initialized: now 1786373079405886 us; error 39 us; skew 500 ppm
I20260810 14:44:39.411419 14187 webserver.cc:460] Webserver started at http://127.15.84.62:40251/ using document root <none> and password file <none>
I20260810 14:44:39.412089 14187 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:39.412184 14187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:39.412396 14187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:39.414616 14187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "ca297f524d8840a4a48eaa6ab559d61b"
format_stamp: "Formatted at 2026-08-10 14:44:39 on dist-test-slave-rmhg"
server_key: "ee0dec624b06ca3b94a1e3be848fba7d"
server_key_iv: "0ff9a83c0d81a64a1cabeeae471ad127"
server_key_version: "encryptionkey@0"
I20260810 14:44:39.415184 14187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "ca297f524d8840a4a48eaa6ab559d61b"
format_stamp: "Formatted at 2026-08-10 14:44:39 on dist-test-slave-rmhg"
server_key: "ee0dec624b06ca3b94a1e3be848fba7d"
server_key_iv: "0ff9a83c0d81a64a1cabeeae471ad127"
server_key_version: "encryptionkey@0"
I20260810 14:44:39.419008 14187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:44:39.421640 14204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:39.422902 14187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:44:39.423056 14187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "ca297f524d8840a4a48eaa6ab559d61b"
format_stamp: "Formatted at 2026-08-10 14:44:39 on dist-test-slave-rmhg"
server_key: "ee0dec624b06ca3b94a1e3be848fba7d"
server_key_iv: "0ff9a83c0d81a64a1cabeeae471ad127"
server_key_version: "encryptionkey@0"
I20260810 14:44:39.423166 14187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:39.437849 14187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:39.438617 14187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:39.438799 14187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:39.446281 14187 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:33677
I20260810 14:44:39.446290 14297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:33677 every 8 connection(s)
I20260810 14:44:39.447311 14187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:39.450603 14299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:39.450922 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 14187
I20260810 14:44:39.451045 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:39.451314 15696 external_mini_cluster.cc:1428] Setting key c427c648612ce011be8bc994aea59057
I20260810 14:44:39.458396 14299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b: Bootstrap starting.
I20260810 14:44:39.461333 14299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:39.462113 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:39.452695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.15.84.62:33677)
0810 14:44:39.453445 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:39.453466 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:39.453515 (+ 49us) server_negotiation.cc:407] Connection header received
0810 14:44:39.454255 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:39.454282 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:39.454856 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:39.455250 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:39.455726 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.457281 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.458219 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.458926 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.459095 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:39.459965 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:39.459972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:39.459989 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:39.460483 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:39.460566 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:39.460594 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:39.460692 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:39.460795 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 14:44:39.461076 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":125,"threads_started":1}
I20260810 14:44:39.462291 14299 log.cc:824] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:39.464394 14299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b: No bootstrap required, opened a new log
I20260810 14:44:39.468171 14299 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca297f524d8840a4a48eaa6ab559d61b" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 33677 } }
I20260810 14:44:39.468436 14299 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:39.468489 14299 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca297f524d8840a4a48eaa6ab559d61b, State: Initialized, Role: FOLLOWER
I20260810 14:44:39.469025 14299 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca297f524d8840a4a48eaa6ab559d61b" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 33677 } }
I20260810 14:44:39.469223 14299 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:39.469301 14299 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:39.469409 14299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:39.470534 14299 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca297f524d8840a4a48eaa6ab559d61b" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 33677 } }
I20260810 14:44:39.470913 14299 leader_election.cc:302] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca297f524d8840a4a48eaa6ab559d61b; no voters:
I20260810 14:44:39.471266 14299 leader_election.cc:288] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:39.471371 14307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:39.471614 14307 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [term 1 LEADER]: Becoming Leader. State: Replica: ca297f524d8840a4a48eaa6ab559d61b, State: Running, Role: LEADER
I20260810 14:44:39.471947 14307 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca297f524d8840a4a48eaa6ab559d61b" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 33677 } }
I20260810 14:44:39.472669 14299 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:39.473735 14308 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca297f524d8840a4a48eaa6ab559d61b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca297f524d8840a4a48eaa6ab559d61b" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 33677 } } }
I20260810 14:44:39.473882 14308 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:39.473735 14311 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca297f524d8840a4a48eaa6ab559d61b. Latest consensus state: current_term: 1 leader_uuid: "ca297f524d8840a4a48eaa6ab559d61b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca297f524d8840a4a48eaa6ab559d61b" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 33677 } } }
I20260810 14:44:39.474184 14311 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:39.474401 14319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:39.477723 14319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:39.483816 14319 catalog_manager.cc:1347] Generated new cluster ID: fac0f8339d90447ebf578571776c2022
I20260810 14:44:39.483901 14319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:39.499765 14319 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:39.501143 14319 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:39.516494 14319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b: Generated new TSK 0
I20260810 14:44:39.517391 14319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:39.528170 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.1:0
--local_ip_for_outbound_sockets=127.15.84.1
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:33677
--builtin_ntp_servers=127.15.84.20:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:44:39.648605 14333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:39.648897 14333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:39.648936 14333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:39.653079 14333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:39.653180 14333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:39.653247 14333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.1
I20260810 14:44:39.658222 14333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.84.1
--webserver_port=0
--tserver_master_addrs=127.15.84.62:33677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.1
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:39.659364 14333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:39.660801 14333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:39.670625 14340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:39.670688 14343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:39.670627 14339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:39.670786 14333 server_base.cc:1034] running on GCE node
I20260810 14:44:39.671499 14333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:39.672123 14333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:39.673295 14333 hybrid_clock.cc:648] HybridClock initialized: now 1786373079673275 us; error 31 us; skew 500 ppm
I20260810 14:44:39.675307 14333 webserver.cc:460] Webserver started at http://127.15.84.1:42777/ using document root <none> and password file <none>
I20260810 14:44:39.675879 14333 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:39.675946 14333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:39.676143 14333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:39.678308 14333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/data/instance:
uuid: "66830ca9fbb3487f9e4bcca907a7f2ae"
format_stamp: "Formatted at 2026-08-10 14:44:39 on dist-test-slave-rmhg"
server_key: "43c0c65892864fe3049408c4a5fc378c"
server_key_iv: "3f7f992b02d617cc2afeef54cc3bbaff"
server_key_version: "encryptionkey@0"
I20260810 14:44:39.678844 14333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance:
uuid: "66830ca9fbb3487f9e4bcca907a7f2ae"
format_stamp: "Formatted at 2026-08-10 14:44:39 on dist-test-slave-rmhg"
server_key: "43c0c65892864fe3049408c4a5fc378c"
server_key_iv: "3f7f992b02d617cc2afeef54cc3bbaff"
server_key_version: "encryptionkey@0"
I20260810 14:44:39.682569 14333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:44:39.685021 14352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:39.686242 14333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:44:39.686363 14333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/wal
uuid: "66830ca9fbb3487f9e4bcca907a7f2ae"
format_stamp: "Formatted at 2026-08-10 14:44:39 on dist-test-slave-rmhg"
server_key: "43c0c65892864fe3049408c4a5fc378c"
server_key_iv: "3f7f992b02d617cc2afeef54cc3bbaff"
server_key_version: "encryptionkey@0"
I20260810 14:44:39.686461 14333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:39.704267 14333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:39.705060 14333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:39.705287 14333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:39.705919 14333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:39.706918 14333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:39.706967 14333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:39.707024 14333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:39.707050 14333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:39.716277 14333 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.1:39933
I20260810 14:44:39.716296 14525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.1:39933 every 8 connection(s)
I20260810 14:44:39.717330 14333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/data/info.pb
I20260810 14:44:39.726569 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 14333
I20260810 14:44:39.726735 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-0/wal/instance
I20260810 14:44:39.727080 15696 external_mini_cluster.cc:1428] Setting key 69eaec72b8ac65c92ebe22ee8fd61da6
I20260810 14:44:39.727242 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:39.719367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:47831 (local address 127.15.84.62:33677)
0810 14:44:39.719534 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:39.719538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:39.720691 (+ 1153us) server_negotiation.cc:407] Connection header received
0810 14:44:39.722173 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:39.722177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:39.722236 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:39.722313 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:39.723323 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.723866 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.725077 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.725368 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.725465 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:39.726706 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:39.726710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:39.726713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:39.726732 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:39.726757 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:39.726768 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:39.726858 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:39.727010 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 14:44:39.727089 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 14:44:39.728492 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:39.719777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:33677 (local address 127.15.84.1:47831)
0810 14:44:39.720503 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:39.720573 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:39.721948 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:39.722506 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:39.722514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:39.722619 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:39.723158 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:39.723171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.723994 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.723997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:39.724944 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:39.724954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.725529 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.725534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:39.725704 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:39.726327 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:44:39.726329 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:39.726374 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:44:39.726444 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:39.726860 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:39.726870 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:39.726877 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:44:39.726993 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 14:44:39.727299 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":138,"threads_started":1}
I20260810 14:44:39.729430 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.2:0
--local_ip_for_outbound_sockets=127.15.84.2
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:33677
--builtin_ntp_servers=127.15.84.20:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:44:39.729949 14526 heartbeater.cc:347] Connected to a master server at 127.15.84.62:33677
I20260810 14:44:39.730252 14526 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:39.730958 14526 heartbeater.cc:511] Master 127.15.84.62:33677 requested a full tablet report, sending...
I20260810 14:44:39.732877 14236 ts_manager.cc:194] Registered new tserver with Master: 66830ca9fbb3487f9e4bcca907a7f2ae (127.15.84.1:39933)
I20260810 14:44:39.734146 14236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.84.1:47831
W20260810 14:44:39.850746 14530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:39.851061 14530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:39.851104 14530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:39.855228 14530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:39.855301 14530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:39.855361 14530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.2
I20260810 14:44:39.860278 14530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.84.2
--webserver_port=0
--tserver_master_addrs=127.15.84.62:33677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.2
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:39.861438 14530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:39.862838 14530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:39.872668 14542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:39.872685 14540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:39.872689 14539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:39.873025 14530 server_base.cc:1034] running on GCE node
I20260810 14:44:39.873564 14530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:39.874253 14530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:39.875413 14530 hybrid_clock.cc:648] HybridClock initialized: now 1786373079875355 us; error 70 us; skew 500 ppm
I20260810 14:44:39.877276 14530 webserver.cc:460] Webserver started at http://127.15.84.2:46547/ using document root <none> and password file <none>
I20260810 14:44:39.877853 14530 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:39.877915 14530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:39.878125 14530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:39.880163 14530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/data/instance:
uuid: "dee9e537c5a84ef1b062fcb8c1d597c3"
format_stamp: "Formatted at 2026-08-10 14:44:39 on dist-test-slave-rmhg"
server_key: "dfdf876c2296362b9a73eeb073e399f2"
server_key_iv: "a29462ade5422692a70777eb0f71aa5c"
server_key_version: "encryptionkey@0"
I20260810 14:44:39.880704 14530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance:
uuid: "dee9e537c5a84ef1b062fcb8c1d597c3"
format_stamp: "Formatted at 2026-08-10 14:44:39 on dist-test-slave-rmhg"
server_key: "dfdf876c2296362b9a73eeb073e399f2"
server_key_iv: "a29462ade5422692a70777eb0f71aa5c"
server_key_version: "encryptionkey@0"
I20260810 14:44:39.884371 14530 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260810 14:44:39.886946 14550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:39.888325 14530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:44:39.888449 14530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/wal
uuid: "dee9e537c5a84ef1b062fcb8c1d597c3"
format_stamp: "Formatted at 2026-08-10 14:44:39 on dist-test-slave-rmhg"
server_key: "dfdf876c2296362b9a73eeb073e399f2"
server_key_iv: "a29462ade5422692a70777eb0f71aa5c"
server_key_version: "encryptionkey@0"
I20260810 14:44:39.888588 14530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:39.901705 14530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:39.902411 14530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:39.902597 14530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:39.903198 14530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:39.904206 14530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:39.904258 14530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:39.904311 14530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:39.904336 14530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:39.913259 14530 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.2:41131
I20260810 14:44:39.913300 14734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.2:41131 every 8 connection(s)
I20260810 14:44:39.914244 14530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/data/info.pb
I20260810 14:44:39.917925 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 14530
I20260810 14:44:39.918077 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/wal/instance
I20260810 14:44:39.918426 15696 external_mini_cluster.cc:1428] Setting key f5f5ad4608bc1c01b059c49a59c9b3d8
I20260810 14:44:39.920849 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.84.3:0
--local_ip_for_outbound_sockets=127.15.84.3
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:33677
--builtin_ntp_servers=127.15.84.20:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:44:39.925299 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:39.916285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:50127 (local address 127.15.84.62:33677)
0810 14:44:39.916432 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:39.916437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:39.917746 (+ 1309us) server_negotiation.cc:407] Connection header received
0810 14:44:39.919564 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:39.919568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:39.919617 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:39.919709 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:39.921001 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.921635 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.923200 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.923536 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.923639 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:39.924779 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:39.924783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:39.924786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:39.924801 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:39.924826 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:39.924835 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:39.924908 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:39.925042 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 14:44:39.925113 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:44:39.926318 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:39.916786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:33677 (local address 127.15.84.2:50127)
0810 14:44:39.917554 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:39.917607 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:39.919301 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:39.919918 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:39.919928 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:39.920043 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:39.920812 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:39.920835 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.921777 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.921781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:39.923060 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:39.923071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:39.923681 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:39.923683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:39.923842 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:39.924483 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:44:39.924485 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:39.924531 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:44:39.924610 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:39.924916 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:39.924921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:39.924926 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:44:39.925027 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 14:44:39.925356 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":128,"threads_started":1}
I20260810 14:44:39.927539 14736 heartbeater.cc:347] Connected to a master server at 127.15.84.62:33677
I20260810 14:44:39.927790 14736 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:39.928351 14736 heartbeater.cc:511] Master 127.15.84.62:33677 requested a full tablet report, sending...
I20260810 14:44:39.929502 14236 ts_manager.cc:194] Registered new tserver with Master: dee9e537c5a84ef1b062fcb8c1d597c3 (127.15.84.2:41131)
I20260810 14:44:39.930097 14236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.84.2:50127
W20260810 14:44:40.039479 14739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:40.039773 14739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:40.039813 14739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:40.043965 14739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:40.044059 14739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:40.044118 14739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.84.3
I20260810 14:44:40.049191 14739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.84.3
--webserver_port=0
--tserver_master_addrs=127.15.84.62:33677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.84.3
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:40.050426 14739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:40.051888 14739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:40.061968 14751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:40.061980 14752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:40.061973 14754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:40.062177 14739 server_base.cc:1034] running on GCE node
I20260810 14:44:40.062768 14739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:40.063491 14739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:40.064702 14739 hybrid_clock.cc:648] HybridClock initialized: now 1786373080064681 us; error 35 us; skew 500 ppm
I20260810 14:44:40.066949 14739 webserver.cc:460] Webserver started at http://127.15.84.3:34497/ using document root <none> and password file <none>
I20260810 14:44:40.067597 14739 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:40.067675 14739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:40.067884 14739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:40.070225 14739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/data/instance:
uuid: "00754949b6e442eb869bad1dc4f57e08"
format_stamp: "Formatted at 2026-08-10 14:44:40 on dist-test-slave-rmhg"
server_key: "1687c9eec22bcfb7d0bf258a7d3d02f7"
server_key_iv: "7533642d86eb356f07d607545c2a9384"
server_key_version: "encryptionkey@0"
I20260810 14:44:40.070827 14739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance:
uuid: "00754949b6e442eb869bad1dc4f57e08"
format_stamp: "Formatted at 2026-08-10 14:44:40 on dist-test-slave-rmhg"
server_key: "1687c9eec22bcfb7d0bf258a7d3d02f7"
server_key_iv: "7533642d86eb356f07d607545c2a9384"
server_key_version: "encryptionkey@0"
I20260810 14:44:40.075178 14739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:44:40.077921 14762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:40.079277 14739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:44:40.079409 14739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/wal
uuid: "00754949b6e442eb869bad1dc4f57e08"
format_stamp: "Formatted at 2026-08-10 14:44:40 on dist-test-slave-rmhg"
server_key: "1687c9eec22bcfb7d0bf258a7d3d02f7"
server_key_iv: "7533642d86eb356f07d607545c2a9384"
server_key_version: "encryptionkey@0"
I20260810 14:44:40.079527 14739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:40.100067 14739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:40.100863 14739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:40.101073 14739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:40.101778 14739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:44:40.102834 14739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:44:40.102886 14739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:40.102944 14739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:44:40.102979 14739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:40.112037 14739 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.3:41557
I20260810 14:44:40.112080 14930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.3:41557 every 8 connection(s)
I20260810 14:44:40.113046 14739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/data/info.pb
I20260810 14:44:40.119139 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 14739
I20260810 14:44:40.119258 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/wal/instance
I20260810 14:44:40.119552 15696 external_mini_cluster.cc:1428] Setting key 3cade3c4e801e59dfa950fa0571728dd
I20260810 14:44:40.122774 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.115052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.3:55179 (local address 127.15.84.62:33677)
0810 14:44:40.115197 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:40.115201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:40.116352 (+ 1151us) server_negotiation.cc:407] Connection header received
0810 14:44:40.117859 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:40.117863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:40.117925 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:40.118005 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:40.119016 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.119608 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.120819 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.121039 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.121132 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.122268 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:40.122272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:40.122275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:40.122287 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:40.122308 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:40.122315 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:40.122376 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:40.122527 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 14:44:40.122594 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:44:40.123725 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.115481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:33677 (local address 127.15.84.3:55179)
0810 14:44:40.116186 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:40.116233 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:40.117618 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:40.118217 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:40.118224 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:40.118321 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:44:40.118859 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.118872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.119746 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.119749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.120686 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.120696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.121223 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.121228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.121393 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.121974 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:44:40.121976 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:44:40.122020 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:44:40.122106 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:44:40.122386 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:44:40.122395 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:44:40.122402 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:44:40.122518 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 14:44:40.122839 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":112,"threads_started":1}
I20260810 14:44:40.125007 14931 heartbeater.cc:347] Connected to a master server at 127.15.84.62:33677
I20260810 14:44:40.125276 14931 heartbeater.cc:464] Registering TS with master...
I20260810 14:44:40.125828 14931 heartbeater.cc:511] Master 127.15.84.62:33677 requested a full tablet report, sending...
I20260810 14:44:40.126955 14236 ts_manager.cc:194] Registered new tserver with Master: 00754949b6e442eb869bad1dc4f57e08 (127.15.84.3:41557)
I20260810 14:44:40.127525 14236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.84.3:55179
I20260810 14:44:40.133818 15696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:44:40.139441 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.135834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.15.84.62:33677)
0810 14:44:40.136000 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:40.136004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:40.136111 (+ 107us) server_negotiation.cc:407] Connection header received
0810 14:44:40.136223 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:40.136225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:40.136270 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:40.136341 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:40.136562 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.137084 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.137923 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.138168 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.138252 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.139053 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:40.139056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:40.139058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:40.139071 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:40.139093 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:40.139100 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:40.139156 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:40.139224 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 14:44:40.139278 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:44:40.142633 14236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:44:40.145701 14236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:44:40.161132 14948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.154857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:39933 (local address 127.0.0.1:46106)
0810 14:44:40.156499 (+ 1642us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:40.156511 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:40.156596 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:40.157064 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:40.157066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:40.157105 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:40.157242 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.157251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.158265 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.158268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.159698 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.159708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.160664 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.160668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.160760 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.160796 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 14:44:40.160881 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 14:44:40.160985 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1586,"thread_start_us":105,"threads_started":1}
I20260810 14:44:40.161309 14952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.155324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.15.84.1:39933)
0810 14:44:40.156077 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:40.156094 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:40.156603 (+ 509us) server_negotiation.cc:407] Connection header received
0810 14:44:40.156695 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:40.156699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:40.156878 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:40.157028 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:40.157366 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.158139 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.159820 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.160531 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.160690 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.160925 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:40.160983 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 14:44:40.161108 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":126,"threads_started":1}
I20260810 14:44:40.162106 14951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.154580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:41557 (local address 127.0.0.1:60424)
0810 14:44:40.156225 (+ 1645us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:40.156236 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:40.156332 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:40.157202 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:40.157205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:40.157222 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:40.157305 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.157310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.158656 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.158659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.160099 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.160111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.161208 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.161211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.161778 (+ 567us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.161807 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 14:44:40.161884 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:44:40.161962 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1587,"mutex_wait_us":118,"thread_start_us":106,"threads_started":1}
I20260810 14:44:40.162688 14954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.155433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.15.84.3:41557)
0810 14:44:40.155841 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:40.155855 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:40.156369 (+ 514us) server_negotiation.cc:407] Connection header received
0810 14:44:40.156452 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:40.156458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:40.156701 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:40.156870 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:40.157411 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.158534 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.160241 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.161044 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.161173 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.161369 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:40.162406 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0810 14:44:40.162519 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":183,"threads_started":1}
I20260810 14:44:40.162986 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.154552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:41131 (local address 127.0.0.1:58848)
0810 14:44:40.155526 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:40.155584 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:40.155734 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:40.156864 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:40.156870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:40.156892 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:40.157048 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.157056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.158696 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.158699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.160101 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.160111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.162266 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.162269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.162716 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.162740 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:44:40.162796 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 14:44:40.162867 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":131,"thread_start_us":110,"threads_started":1}
I20260810 14:44:40.163774 14950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.154724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58848 (local address 127.15.84.2:41131)
0810 14:44:40.156324 (+ 1600us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:40.156341 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:40.156364 (+ 23us) server_negotiation.cc:407] Connection header received
0810 14:44:40.156452 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:40.156458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:40.156695 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:40.156871 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:40.157202 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.158489 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.161452 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.162139 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.163270 (+ 1131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.163478 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:40.163532 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 14:44:40.163632 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1457,"thread_start_us":152,"threads_started":1}
I20260810 14:44:40.164976 14422 tablet_service.cc:1467] Processing CreateTablet for tablet 908457f9487341b2832e73ab63606cba (DEFAULT_TABLE table=test-table [id=678682c2c1ef41eb815bc50b0ca5e146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:40.165760 14836 tablet_service.cc:1467] Processing CreateTablet for tablet 908457f9487341b2832e73ab63606cba (DEFAULT_TABLE table=test-table [id=678682c2c1ef41eb815bc50b0ca5e146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:40.166366 14422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 908457f9487341b2832e73ab63606cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:40.166143 14619 tablet_service.cc:1467] Processing CreateTablet for tablet 908457f9487341b2832e73ab63606cba (DEFAULT_TABLE table=test-table [id=678682c2c1ef41eb815bc50b0ca5e146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:44:40.166867 14836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 908457f9487341b2832e73ab63606cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:40.167119 14619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 908457f9487341b2832e73ab63606cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:40.173441 14955 tablet_bootstrap.cc:492] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3: Bootstrap starting.
I20260810 14:44:40.174986 14956 tablet_bootstrap.cc:492] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae: Bootstrap starting.
I20260810 14:44:40.175822 14955 tablet_bootstrap.cc:654] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:40.176553 14955 log.cc:824] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:40.176817 14956 tablet_bootstrap.cc:654] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:40.177529 14957 tablet_bootstrap.cc:492] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08: Bootstrap starting.
I20260810 14:44:40.177708 14956 log.cc:824] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:40.178894 14955 tablet_bootstrap.cc:492] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3: No bootstrap required, opened a new log
I20260810 14:44:40.179102 14955 ts_tablet_manager.cc:1397] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:44:40.179517 14956 tablet_bootstrap.cc:492] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae: No bootstrap required, opened a new log
I20260810 14:44:40.179711 14956 ts_tablet_manager.cc:1397] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 14:44:40.179853 14957 tablet_bootstrap.cc:654] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:40.180689 14957 log.cc:824] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:40.182927 14957 tablet_bootstrap.cc:492] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08: No bootstrap required, opened a new log
I20260810 14:44:40.183138 14957 ts_tablet_manager.cc:1397] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:44:40.183347 14956 raft_consensus.cc:348] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 41131 } } peers { permanent_uuid: "00754949b6e442eb869bad1dc4f57e08" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 41557 } } peers { permanent_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 39933 } }
I20260810 14:44:40.183660 14956 raft_consensus.cc:374] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:40.183504 14955 raft_consensus.cc:348] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 41131 } } peers { permanent_uuid: "00754949b6e442eb869bad1dc4f57e08" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 41557 } } peers { permanent_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 39933 } }
I20260810 14:44:40.183799 14955 raft_consensus.cc:374] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:40.183748 14956 raft_consensus.cc:729] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66830ca9fbb3487f9e4bcca907a7f2ae, State: Initialized, Role: FOLLOWER
I20260810 14:44:40.183869 14955 raft_consensus.cc:729] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee9e537c5a84ef1b062fcb8c1d597c3, State: Initialized, Role: FOLLOWER
I20260810 14:44:40.184435 14956 consensus_queue.cc:260] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 41131 } } peers { permanent_uuid: "00754949b6e442eb869bad1dc4f57e08" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 41557 } } peers { permanent_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 39933 } }
I20260810 14:44:40.184342 14955 consensus_queue.cc:260] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 41131 } } peers { permanent_uuid: "00754949b6e442eb869bad1dc4f57e08" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 41557 } } peers { permanent_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 39933 } }
I20260810 14:44:40.185607 14956 ts_tablet_manager.cc:1428] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 14:44:40.185758 14955 ts_tablet_manager.cc:1428] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 14:44:40.185964 14526 heartbeater.cc:503] Master 127.15.84.62:33677 was elected leader, sending a full tablet report...
I20260810 14:44:40.186151 14736 heartbeater.cc:503] Master 127.15.84.62:33677 was elected leader, sending a full tablet report...
I20260810 14:44:40.187384 14957 raft_consensus.cc:348] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 41131 } } peers { permanent_uuid: "00754949b6e442eb869bad1dc4f57e08" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 41557 } } peers { permanent_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 39933 } }
I20260810 14:44:40.187620 14957 raft_consensus.cc:374] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:40.187697 14957 raft_consensus.cc:729] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00754949b6e442eb869bad1dc4f57e08, State: Initialized, Role: FOLLOWER
I20260810 14:44:40.188122 14957 consensus_queue.cc:260] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 41131 } } peers { permanent_uuid: "00754949b6e442eb869bad1dc4f57e08" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 41557 } } peers { permanent_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 39933 } }
I20260810 14:44:40.188825 14931 heartbeater.cc:503] Master 127.15.84.62:33677 was elected leader, sending a full tablet report...
I20260810 14:44:40.189078 14957 ts_tablet_manager.cc:1428] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260810 14:44:40.218616 14527 tablet.cc:2369] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:44:40.364145 14932 tablet.cc:2369] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:44:40.415475 14737 tablet.cc:2369] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:44:40.516872 14969 raft_consensus.cc:482] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:44:40.517225 14969 raft_consensus.cc:504] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 41131 } } peers { permanent_uuid: "00754949b6e442eb869bad1dc4f57e08" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 41557 } } peers { permanent_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 39933 } }
I20260810 14:44:40.518035 14964 raft_consensus.cc:482] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:44:40.518246 14964 raft_consensus.cc:504] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 41131 } } peers { permanent_uuid: "00754949b6e442eb869bad1dc4f57e08" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 41557 } } peers { permanent_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 39933 } }
I20260810 14:44:40.519057 14969 leader_election.cc:288] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00754949b6e442eb869bad1dc4f57e08 (127.15.84.3:41557), 66830ca9fbb3487f9e4bcca907a7f2ae (127.15.84.1:39933)
I20260810 14:44:40.520146 14964 leader_election.cc:288] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dee9e537c5a84ef1b062fcb8c1d597c3 (127.15.84.2:41131), 00754949b6e442eb869bad1dc4f57e08 (127.15.84.3:41557)
I20260810 14:44:40.525288 14954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.519388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:37609 (local address 127.15.84.3:41557)
0810 14:44:40.519919 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:40.519923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:40.519938 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:44:40.520010 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:40.520014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:40.520061 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:40.520134 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:40.520596 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.521443 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.523078 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.523685 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.523770 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.523844 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:40.525009 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0810 14:44:40.525101 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450}
I20260810 14:44:40.525511 14976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.519684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:41557 (local address 127.15.84.1:44489)
0810 14:44:40.520502 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:40.520514 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:40.520580 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:40.520935 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:40.520938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:40.520954 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:40.521075 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.521082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.522103 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.522107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.523754 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.523763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.525026 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.525030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.525123 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.525145 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:44:40.525250 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 14:44:40.525330 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":479,"thread_start_us":112,"threads_started":1}
I20260810 14:44:40.525911 14950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.520006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:49299 (local address 127.15.84.2:41131)
0810 14:44:40.520109 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:40.520113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:40.520320 (+ 207us) server_negotiation.cc:407] Connection header received
0810 14:44:40.520509 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:40.520512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:40.520558 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:40.520602 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:40.520984 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.521813 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.523885 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.524622 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.524736 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.524798 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:40.525720 (+ 922us) server_negotiation.cc:299] Negotiation successful
0810 14:44:40.525803 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":44}
I20260810 14:44:40.525799 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.518923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.3:41557 (local address 127.15.84.2:37609)
0810 14:44:40.519570 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:40.519585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:40.519683 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:40.520259 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:40.520263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:40.520283 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:40.520458 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.520466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.521591 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.521595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.522922 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.522934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.523947 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.523951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.524869 (+ 918us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.524886 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 14:44:40.525526 (+ 640us) client_negotiation.cc:240] Negotiation successful
0810 14:44:40.525627 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":67,"spinlock_wait_cycles":2432,"thread_start_us":89,"threads_started":1}
I20260810 14:44:40.526048 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.519622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.2:41131 (local address 127.15.84.1:49299)
0810 14:44:40.520247 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:40.520260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:40.520380 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:40.520721 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:40.520723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:40.520737 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:40.520861 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.520868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.521951 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.521954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.523303 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.523313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.524772 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.524776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.524882 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.524908 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:44:40.525836 (+ 928us) client_negotiation.cc:240] Negotiation successful
0810 14:44:40.525904 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":26,"thread_start_us":60,"threads_started":1}
I20260810 14:44:40.526362 14981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.519919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.1:44489 (local address 127.15.84.3:41557)
0810 14:44:40.520569 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:40.520579 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:40.520591 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:44:40.520699 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:40.520703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:40.520745 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:40.520815 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:40.521231 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.521994 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.523977 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.524627 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.524735 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.524796 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:40.526146 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0810 14:44:40.526230 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":126,"threads_started":1}
I20260810 14:44:40.526769 14656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908457f9487341b2832e73ab63606cba" candidate_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" is_pre_election: true
I20260810 14:44:40.526842 14867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908457f9487341b2832e73ab63606cba" candidate_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00754949b6e442eb869bad1dc4f57e08" is_pre_election: true
I20260810 14:44:40.527047 14656 raft_consensus.cc:2454] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66830ca9fbb3487f9e4bcca907a7f2ae in term 0.
I20260810 14:44:40.527164 14867 raft_consensus.cc:2454] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66830ca9fbb3487f9e4bcca907a7f2ae in term 0.
I20260810 14:44:40.526855 14868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908457f9487341b2832e73ab63606cba" candidate_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00754949b6e442eb869bad1dc4f57e08" is_pre_election: true
I20260810 14:44:40.527490 14868 raft_consensus.cc:2454] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee9e537c5a84ef1b062fcb8c1d597c3 in term 0.
I20260810 14:44:40.527644 14357 leader_election.cc:302] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66830ca9fbb3487f9e4bcca907a7f2ae, dee9e537c5a84ef1b062fcb8c1d597c3; no voters:
I20260810 14:44:40.528009 14964 raft_consensus.cc:2781] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:44:40.527987 14553 leader_election.cc:302] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00754949b6e442eb869bad1dc4f57e08, dee9e537c5a84ef1b062fcb8c1d597c3; no voters:
I20260810 14:44:40.528101 14964 raft_consensus.cc:482] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:44:40.528144 14964 raft_consensus.cc:3037] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:40.528259 14952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.519104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.84.2:52057 (local address 127.15.84.1:39933)
0810 14:44:40.519244 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:40.519248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:40.519815 (+ 567us) server_negotiation.cc:407] Connection header received
0810 14:44:40.520013 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:40.520016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:40.520069 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:40.520116 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:44:40.523202 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.523871 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.526963 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.527544 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.527647 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.527715 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:40.528075 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 14:44:40.528154 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:44:40.528306 14969 raft_consensus.cc:2781] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:44:40.528376 14969 raft_consensus.cc:482] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:44:40.528422 14969 raft_consensus.cc:3037] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:40.528697 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.518923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:39933 (local address 127.15.84.2:52057)
0810 14:44:40.519727 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:40.519743 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:44:40.519890 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:44:40.520278 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:44:40.520281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:44:40.520300 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:44:40.523064 (+ 2764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.523073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.524891 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.524893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.526007 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:44:40.526017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.527834 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.527837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:44:40.527933 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.527958 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:44:40.528431 (+ 473us) client_negotiation.cc:240] Negotiation successful
0810 14:44:40.528515 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":326,"thread_start_us":68,"threads_started":1}
I20260810 14:44:40.529604 14969 raft_consensus.cc:504] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 41131 } } peers { permanent_uuid: "00754949b6e442eb869bad1dc4f57e08" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 41557 } } peers { permanent_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 39933 } }
I20260810 14:44:40.529605 14964 raft_consensus.cc:504] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 41131 } } peers { permanent_uuid: "00754949b6e442eb869bad1dc4f57e08" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 41557 } } peers { permanent_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 39933 } }
I20260810 14:44:40.529631 14453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908457f9487341b2832e73ab63606cba" candidate_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" is_pre_election: true
I20260810 14:44:40.529920 14453 raft_consensus.cc:2379] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dee9e537c5a84ef1b062fcb8c1d597c3 in current term 1: Already voted for candidate 66830ca9fbb3487f9e4bcca907a7f2ae in this term.
I20260810 14:44:40.530006 14969 leader_election.cc:288] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [CANDIDATE]: Term 1 election: Requested vote from peers 00754949b6e442eb869bad1dc4f57e08 (127.15.84.3:41557), 66830ca9fbb3487f9e4bcca907a7f2ae (127.15.84.1:39933)
I20260810 14:44:40.530006 14964 leader_election.cc:288] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [CANDIDATE]: Term 1 election: Requested vote from peers dee9e537c5a84ef1b062fcb8c1d597c3 (127.15.84.2:41131), 00754949b6e442eb869bad1dc4f57e08 (127.15.84.3:41557)
I20260810 14:44:40.530427 14656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908457f9487341b2832e73ab63606cba" candidate_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3"
I20260810 14:44:40.530552 14453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908457f9487341b2832e73ab63606cba" candidate_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae"
I20260810 14:44:40.530681 14656 raft_consensus.cc:2379] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 66830ca9fbb3487f9e4bcca907a7f2ae in current term 1: Already voted for candidate dee9e537c5a84ef1b062fcb8c1d597c3 in this term.
I20260810 14:44:40.530716 14453 raft_consensus.cc:2379] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dee9e537c5a84ef1b062fcb8c1d597c3 in current term 1: Already voted for candidate 66830ca9fbb3487f9e4bcca907a7f2ae in this term.
I20260810 14:44:40.531253 14868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908457f9487341b2832e73ab63606cba" candidate_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00754949b6e442eb869bad1dc4f57e08"
I20260810 14:44:40.531301 14867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "908457f9487341b2832e73ab63606cba" candidate_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00754949b6e442eb869bad1dc4f57e08"
I20260810 14:44:40.531401 14868 raft_consensus.cc:3037] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:40.531868 14355 leader_election.cc:302] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66830ca9fbb3487f9e4bcca907a7f2ae; no voters: 00754949b6e442eb869bad1dc4f57e08, dee9e537c5a84ef1b062fcb8c1d597c3
I20260810 14:44:40.532037 14964 raft_consensus.cc:2726] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 14:44:40.532769 14868 raft_consensus.cc:2454] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee9e537c5a84ef1b062fcb8c1d597c3 in term 1.
I20260810 14:44:40.533123 14553 leader_election.cc:302] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00754949b6e442eb869bad1dc4f57e08, dee9e537c5a84ef1b062fcb8c1d597c3; no voters: 66830ca9fbb3487f9e4bcca907a7f2ae
I20260810 14:44:40.533349 14969 raft_consensus.cc:2781] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:40.533636 14969 raft_consensus.cc:686] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 1 LEADER]: Becoming Leader. State: Replica: dee9e537c5a84ef1b062fcb8c1d597c3, State: Running, Role: LEADER
I20260810 14:44:40.534085 14969 consensus_queue.cc:237] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 41131 } } peers { permanent_uuid: "00754949b6e442eb869bad1dc4f57e08" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 41557 } } peers { permanent_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 39933 } }
I20260810 14:44:40.538002 14233 catalog_manager.cc:5495] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to dee9e537c5a84ef1b062fcb8c1d597c3 (127.15.84.2). New cstate: current_term: 1 leader_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee9e537c5a84ef1b062fcb8c1d597c3" member_type: VOTER last_known_addr { host: "127.15.84.2" port: 41131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00754949b6e442eb869bad1dc4f57e08" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 41557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20260810 14:44:40.615274 14950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.611730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.15.84.2:41131)
0810 14:44:40.611866 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:40.611870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:40.611884 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:44:40.611986 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:40.611989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:40.612050 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:40.612140 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:44:40.612334 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.612845 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.613747 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:40.613984 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:40.614091 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:40.614370 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:44:40.614833 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:44:40.614954 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:40.615062 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 14:44:40.615126 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:44:40.621403 14868 raft_consensus.cc:1261] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08 [term 1 FOLLOWER]: Refusing update from remote peer dee9e537c5a84ef1b062fcb8c1d597c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:44:40.621404 14453 raft_consensus.cc:1261] T 908457f9487341b2832e73ab63606cba P 66830ca9fbb3487f9e4bcca907a7f2ae [term 1 FOLLOWER]: Refusing update from remote peer dee9e537c5a84ef1b062fcb8c1d597c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:44:40.622074 14983 consensus_queue.cc:1035] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66830ca9fbb3487f9e4bcca907a7f2ae" member_type: VOTER last_known_addr { host: "127.15.84.1" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:44:40.622269 14969 consensus_queue.cc:1035] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00754949b6e442eb869bad1dc4f57e08" member_type: VOTER last_known_addr { host: "127.15.84.3" port: 41557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:44:40.638957 14233 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:44:40.639163 14233 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:44:40.642035 14233 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca297f524d8840a4a48eaa6ab559d61b: Sending DeleteTablet for 3 replicas of tablet 908457f9487341b2832e73ab63606cba
I20260810 14:44:40.642933 14619 tablet_service.cc:1514] Processing DeleteTablet for tablet 908457f9487341b2832e73ab63606cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:40 UTC) from {username='slave'} at 127.0.0.1:58848
I20260810 14:44:40.643154 14836 tablet_service.cc:1514] Processing DeleteTablet for tablet 908457f9487341b2832e73ab63606cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:40 UTC) from {username='slave'} at 127.0.0.1:60424
I20260810 14:44:40.643461 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 14333
I20260810 14:44:40.643538 14422 tablet_service.cc:1514] Processing DeleteTablet for tablet 908457f9487341b2832e73ab63606cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:44:40 UTC) from {username='slave'} at 127.0.0.1:46106
I20260810 14:44:40.650859 15002 tablet_replica.cc:330] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3: stopping tablet replica
I20260810 14:44:40.651324 15002 raft_consensus.cc:2229] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:44:40.651778 15002 raft_consensus.cc:2258] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:44:40.653702 15003 tablet_replica.cc:330] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08: stopping tablet replica
I20260810 14:44:40.654116 15003 raft_consensus.cc:2229] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:44:40.654608 15003 raft_consensus.cc:2258] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:44:40.656390 15003 ts_tablet_manager.cc:1905] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:44:40.658006 15002 ts_tablet_manager.cc:1905] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:44:40.659363 15003 ts_tablet_manager.cc:1918] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:44:40.659471 15003 log.cc:1196] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-2/wal/wals/908457f9487341b2832e73ab63606cba
I20260810 14:44:40.659773 15003 ts_tablet_manager.cc:1939] T 908457f9487341b2832e73ab63606cba P 00754949b6e442eb869bad1dc4f57e08: Deleting consensus metadata
I20260810 14:44:40.661281 14208 catalog_manager.cc:4841] TS 00754949b6e442eb869bad1dc4f57e08 (127.15.84.3:41557): tablet 908457f9487341b2832e73ab63606cba (table test-table [id=678682c2c1ef41eb815bc50b0ca5e146]) successfully deleted
I20260810 14:44:40.661346 15002 ts_tablet_manager.cc:1918] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:44:40.661610 15002 log.cc:1196] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3: Deleting WAL directory at /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372819317171-15696-0/minicluster-data/ts-1/wal/wals/908457f9487341b2832e73ab63606cba
I20260810 14:44:40.662001 15002 ts_tablet_manager.cc:1939] T 908457f9487341b2832e73ab63606cba P dee9e537c5a84ef1b062fcb8c1d597c3: Deleting consensus metadata
I20260810 14:44:40.663270 14213 catalog_manager.cc:4841] TS dee9e537c5a84ef1b062fcb8c1d597c3 (127.15.84.2:41131): tablet 908457f9487341b2832e73ab63606cba (table test-table [id=678682c2c1ef41eb815bc50b0ca5e146]) successfully deleted
W20260810 14:44:40.664453 14207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.84.1:39933 (error 108)
I20260810 14:44:40.664593 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 14530
I20260810 14:44:40.676105 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:40.669807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.1:39933 (local address 127.0.0.1:46110)
0810 14:44:40.675912 (+ 6105us) negotiation.cc:107] Waiting for socket to connect
0810 14:44:40.675998 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.1:39933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6014,"thread_start_us":2963,"threads_started":1}
W20260810 14:44:40.676445 14207 catalog_manager.cc:4584] TS 66830ca9fbb3487f9e4bcca907a7f2ae (127.15.84.1:39933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 908457f9487341b2832e73ab63606cba: Network error: Client connection negotiation failed: client connection to 127.15.84.1:39933: connect: Connection refused (error 111)
I20260810 14:44:40.685140 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 14739
I20260810 14:44:40.700948 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 14187
2026-08-10T14:44:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1476 ms)
[----------] 6 tests from EncryptionPolicyTest (11838 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:40 dist-test-slave-rmhg krb5kdc[15014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:40 dist-test-slave-rmhg krb5kdc[15014](info): set up 2 sockets
Aug 10 14:44:40 dist-test-slave-rmhg krb5kdc[15014](info): commencing operation
krb5kdc: starting...
W20260810 14:44:41.854429 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:41 dist-test-slave-rmhg krb5kdc[15014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:44:41.996775 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:42.114027 15035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:42.114337 15035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:42.114380 15035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:42.118538 15035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:42.118613 15035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:42.118633 15035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:42.118650 15035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:42.118680 15035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:42.124383 15035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39639
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:42.125618 15035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:42.126685 15035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:42.135680 15046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:42.135681 15042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:42.135754 15035 server_base.cc:1034] running on GCE node
W20260810 14:44:42.135681 15040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:42.136577 15035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:42.137885 15035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:42.139083 15035 hybrid_clock.cc:648] HybridClock initialized: now 1786373082139058 us; error 39 us; skew 500 ppm
Aug 10 14:44:42 dist-test-slave-rmhg krb5kdc[15014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:42.142261 15035 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:44:42.143265 15035 webserver.cc:460] Webserver started at http://127.0.0.1:32837/ using document root <none> and password file <none>
I20260810 14:44:42.143849 15035 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:42.143924 15035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:42.144148 15035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:42.146386 15035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "88f373bc403f4490adf220cac3955eaa"
format_stamp: "Formatted at 2026-08-10 14:44:42 on dist-test-slave-rmhg"
server_key: "bc7c806236f5fb6fc654870124df5755"
server_key_iv: "aa62f0c52f8e60d677c6065862fa9dfd"
server_key_version: "encryptionkey@0"
I20260810 14:44:42.146978 15035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "88f373bc403f4490adf220cac3955eaa"
format_stamp: "Formatted at 2026-08-10 14:44:42 on dist-test-slave-rmhg"
server_key: "bc7c806236f5fb6fc654870124df5755"
server_key_iv: "aa62f0c52f8e60d677c6065862fa9dfd"
server_key_version: "encryptionkey@0"
I20260810 14:44:42.150921 15035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:44:42.153496 15057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:42.154815 15035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:44:42.154942 15035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "88f373bc403f4490adf220cac3955eaa"
format_stamp: "Formatted at 2026-08-10 14:44:42 on dist-test-slave-rmhg"
server_key: "bc7c806236f5fb6fc654870124df5755"
server_key_iv: "aa62f0c52f8e60d677c6065862fa9dfd"
server_key_version: "encryptionkey@0"
I20260810 14:44:42.155048 15035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:42.179971 15035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:42.183079 15035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:42.183320 15035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:42.190819 15035 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39639
I20260810 14:44:42.190826 15131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39639 every 8 connection(s)
I20260810 14:44:42.191870 15035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:42.194509 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 15035
I20260810 14:44:42.194659 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:42.194978 15696 external_mini_cluster.cc:1428] Setting key 9656aa481cdfd145ec7ead2b0ef57d7f
I20260810 14:44:42.195082 15132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:42.202518 15132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa: Bootstrap starting.
Aug 10 14:44:42 dist-test-slave-rmhg krb5kdc[15014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 14:44:42.205618 15132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:42.206540 15132 log.cc:824] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:42.209072 15132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa: No bootstrap required, opened a new log
I20260810 14:44:42.211292 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:42.196471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40139 (local address 127.0.0.1:39639)
0810 14:44:42.196923 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:42.196939 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:42.196977 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:44:42.197844 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:42.197866 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:42.198256 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:42.198565 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:42.199537 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:42.200575 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:42.201834 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:42.202484 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:42.202712 (+ 228us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:42.205839 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:42.205854 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:42.205870 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:42.205908 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:42.207892 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:42.208511 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:42.208515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:42.208520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:42.208671 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:42.209092 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:42.209095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:42.209097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:42.209651 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:42.209831 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:42.210221 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 14:44:42.210469 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":145,"threads_started":1}
I20260810 14:44:42.213241 15132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f373bc403f4490adf220cac3955eaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39639 } }
I20260810 14:44:42.213562 15132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:42.213629 15132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f373bc403f4490adf220cac3955eaa, State: Initialized, Role: FOLLOWER
I20260810 14:44:42.214097 15132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f373bc403f4490adf220cac3955eaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39639 } }
I20260810 14:44:42.214265 15132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:42.214334 15132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:42.214439 15132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:42.214581 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:44:42.215433 15132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f373bc403f4490adf220cac3955eaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39639 } }
I20260810 14:44:42.215785 15132 leader_election.cc:302] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88f373bc403f4490adf220cac3955eaa; no voters:
I20260810 14:44:42.216130 15132 leader_election.cc:288] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:42.216241 15150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:42.216936 15150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88f373bc403f4490adf220cac3955eaa, State: Running, Role: LEADER
I20260810 14:44:42.217372 15150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f373bc403f4490adf220cac3955eaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39639 } }
I20260810 14:44:42.217484 15132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:42.219264 15151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f373bc403f4490adf220cac3955eaa. Latest consensus state: current_term: 1 leader_uuid: "88f373bc403f4490adf220cac3955eaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f373bc403f4490adf220cac3955eaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39639 } } }
I20260810 14:44:42.219413 15151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:42.220079 15174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:42.221095 15153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88f373bc403f4490adf220cac3955eaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f373bc403f4490adf220cac3955eaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39639 } } }
I20260810 14:44:42.221251 15153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:42.223627 15174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:42.225441 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:42.216699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33951 (local address 127.0.0.1:39639)
0810 14:44:42.216820 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:42.216825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:42.217109 (+ 284us) server_negotiation.cc:407] Connection header received
0810 14:44:42.217173 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:42.217177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:42.217227 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:42.217293 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:42.218230 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:42.219023 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:42.220084 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:42.220427 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:42.220507 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:42.222309 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:42.222313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:42.222315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:42.222342 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:42.223685 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:42.224170 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:42.224174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:42.224176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:42.224255 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:42.224572 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:42.224575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:42.224577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:42.224771 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:42.224904 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:42.225206 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 14:44:42.225300 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 14:44:42.230695 15174 catalog_manager.cc:1347] Generated new cluster ID: a2fedc8b86b84999aa006da439ce9680
I20260810 14:44:42.230782 15174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:42.254999 15174 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:42.256006 15174 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:42.266525 15174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88f373bc403f4490adf220cac3955eaa: Generated new TSK 0
I20260810 14:44:42.267270 15174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:42.275861 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 15035
W20260810 14:44:42.297003 15006 connection.cc:535] client connection to 127.0.0.1:39639 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39639): Connection reset by peer (error 104)
2026-08-10T14:44:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1581 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:42 dist-test-slave-rmhg krb5kdc[15182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:42 dist-test-slave-rmhg krb5kdc[15182](info): set up 2 sockets
Aug 10 14:44:42 dist-test-slave-rmhg krb5kdc[15182](info): commencing operation
krb5kdc: starting...
W20260810 14:44:43.437364 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:43 dist-test-slave-rmhg krb5kdc[15182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:44:43.570003 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:43.695194 15198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:43.695508 15198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:43.695577 15198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:43.699637 15198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:43.699712 15198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:43.699738 15198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:43.699759 15198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:43.699780 15198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:43.705279 15198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39655
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:43.706468 15198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:43.707568 15198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:43.716284 15207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:43.716284 15211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:43.716285 15214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:43.716604 15198 server_base.cc:1034] running on GCE node
I20260810 14:44:43.717257 15198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:43.718422 15198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:43.719592 15198 hybrid_clock.cc:648] HybridClock initialized: now 1786373083719572 us; error 33 us; skew 500 ppm
Aug 10 14:44:43 dist-test-slave-rmhg krb5kdc[15182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:43.722401 15198 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:44:43.723385 15198 webserver.cc:460] Webserver started at http://127.0.0.1:38367/ using document root <none> and password file <none>
I20260810 14:44:43.723949 15198 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:43.724015 15198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:43.724246 15198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:43.726444 15198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "d5f76bb9ccea45e58bd882803d038e6c"
format_stamp: "Formatted at 2026-08-10 14:44:43 on dist-test-slave-rmhg"
server_key: "299d78405a1eb8677f9275c10dedf99d"
server_key_iv: "328fe0161fae7ba4657da43e775b887d"
server_key_version: "encryptionkey@0"
I20260810 14:44:43.727011 15198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "d5f76bb9ccea45e58bd882803d038e6c"
format_stamp: "Formatted at 2026-08-10 14:44:43 on dist-test-slave-rmhg"
server_key: "299d78405a1eb8677f9275c10dedf99d"
server_key_iv: "328fe0161fae7ba4657da43e775b887d"
server_key_version: "encryptionkey@0"
I20260810 14:44:43.730988 15198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 14:44:43.733543 15224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:43.734815 15198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:44:43.734936 15198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "d5f76bb9ccea45e58bd882803d038e6c"
format_stamp: "Formatted at 2026-08-10 14:44:43 on dist-test-slave-rmhg"
server_key: "299d78405a1eb8677f9275c10dedf99d"
server_key_iv: "328fe0161fae7ba4657da43e775b887d"
server_key_version: "encryptionkey@0"
I20260810 14:44:43.735061 15198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:43.750936 15198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:43.753962 15198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:43.754156 15198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:43.761389 15198 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39655
I20260810 14:44:43.761401 15298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39655 every 8 connection(s)
I20260810 14:44:43.762475 15198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:43.765473 15299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:43.768205 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 15198
I20260810 14:44:43.768373 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:43.768688 15696 external_mini_cluster.cc:1428] Setting key 03b7526a7034924d55b85feb27c7d3b7
I20260810 14:44:43.771998 15299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c: Bootstrap starting.
I20260810 14:44:43.774730 15299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:43.775586 15299 log.cc:824] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:43.777813 15299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c: No bootstrap required, opened a new log
I20260810 14:44:43.778131 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:43.770199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58645 (local address 127.0.0.1:39655)
0810 14:44:43.770670 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:43.770689 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:43.770730 (+ 41us) server_negotiation.cc:407] Connection header received
0810 14:44:43.771397 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:43.771417 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:43.771791 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:43.772100 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:43.772478 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:43.773525 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:43.774705 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:43.775248 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:43.775367 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:43.776283 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:43.776290 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:43.776308 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:43.776734 (+ 426us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:43.776791 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:43.776815 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:43.776917 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:43.777128 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 14:44:43.777332 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":134,"threads_started":1}
I20260810 14:44:43.781136 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:44:43.781685 15299 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f76bb9ccea45e58bd882803d038e6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39655 } }
I20260810 14:44:43.781996 15299 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:43.782063 15299 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f76bb9ccea45e58bd882803d038e6c, State: Initialized, Role: FOLLOWER
I20260810 14:44:43.782562 15299 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f76bb9ccea45e58bd882803d038e6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39655 } }
I20260810 14:44:43.782749 15299 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:43.782823 15299 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:43.782927 15299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:43.784056 15299 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f76bb9ccea45e58bd882803d038e6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39655 } }
I20260810 14:44:43.784443 15299 leader_election.cc:302] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5f76bb9ccea45e58bd882803d038e6c; no voters:
I20260810 14:44:43.784823 15299 leader_election.cc:288] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:43.784914 15312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:43.785140 15312 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [term 1 LEADER]: Becoming Leader. State: Replica: d5f76bb9ccea45e58bd882803d038e6c, State: Running, Role: LEADER
I20260810 14:44:43.785495 15312 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f76bb9ccea45e58bd882803d038e6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39655 } }
I20260810 14:44:43.786242 15299 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:43.787278 15313 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5f76bb9ccea45e58bd882803d038e6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f76bb9ccea45e58bd882803d038e6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39655 } } }
I20260810 14:44:43.787422 15313 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:43.787762 15314 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5f76bb9ccea45e58bd882803d038e6c. Latest consensus state: current_term: 1 leader_uuid: "d5f76bb9ccea45e58bd882803d038e6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f76bb9ccea45e58bd882803d038e6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39655 } } }
I20260810 14:44:43.787878 15314 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:43.788836 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:43.783535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50025 (local address 127.0.0.1:39655)
0810 14:44:43.783655 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:43.783660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:43.783871 (+ 211us) server_negotiation.cc:407] Connection header received
0810 14:44:43.783955 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:43.783958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:43.784010 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:43.784102 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:43.784345 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:43.785135 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:43.786319 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:43.786685 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:43.786793 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:43.788018 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:43.788022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:43.788025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:43.788039 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:43.788069 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:43.788085 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:43.788170 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:43.788669 (+ 499us) server_negotiation.cc:299] Negotiation successful
0810 14:44:43.788709 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 14:44:43.791517 15343 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:44:43.791595 15343 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:44:43.791676 15322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:43.792449 15322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:43.798702 15322 catalog_manager.cc:1347] Generated new cluster ID: 1de50e890e694c6cabbf22f521e16d1d
I20260810 14:44:43.798807 15322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:43.817281 15322 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:43.818650 15322 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:43.827956 15322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5f76bb9ccea45e58bd882803d038e6c: Generated new TSK 0
I20260810 14:44:43.828868 15322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:43.837950 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 15198
W20260810 14:44:43.854229 15179 connection.cc:535] client connection to 127.0.0.1:39655 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39655): Connection reset by peer (error 104)
2026-08-10T14:44:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1557 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:43 dist-test-slave-rmhg krb5kdc[15352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:43 dist-test-slave-rmhg krb5kdc[15352](info): set up 2 sockets
Aug 10 14:44:43 dist-test-slave-rmhg krb5kdc[15352](info): commencing operation
krb5kdc: starting...
W20260810 14:44:44.995952 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:45 dist-test-slave-rmhg krb5kdc[15352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:44:45.124578 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40039
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:45.243048 15370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:45.243351 15370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:45.243394 15370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:45.247987 15370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:45.248077 15370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:45.248095 15370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:45.248111 15370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:45.248126 15370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:45.253795 15370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40039
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:45.254932 15370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:45.255975 15370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:45.264793 15381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:45.264830 15378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:45.264935 15370 server_base.cc:1034] running on GCE node
W20260810 14:44:45.264819 15383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:45.265698 15370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:45.266878 15370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:45.268041 15370 hybrid_clock.cc:648] HybridClock initialized: now 1786373085268024 us; error 34 us; skew 500 ppm
Aug 10 14:44:45 dist-test-slave-rmhg krb5kdc[15352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:45.274660 15370 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:44:45.275702 15370 webserver.cc:460] Webserver started at http://127.0.0.1:40345/ using document root <none> and password file <none>
I20260810 14:44:45.276279 15370 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:45.276352 15370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:45.276577 15370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:45.278781 15370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "b5a005977fdf4982ab8f348919538250"
format_stamp: "Formatted at 2026-08-10 14:44:45 on dist-test-slave-rmhg"
server_key: "24295571305c89fe77d8ed96e88f57c7"
server_key_iv: "8329b6afac2ded2a4c8e871522177f3e"
server_key_version: "encryptionkey@0"
I20260810 14:44:45.279333 15370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "b5a005977fdf4982ab8f348919538250"
format_stamp: "Formatted at 2026-08-10 14:44:45 on dist-test-slave-rmhg"
server_key: "24295571305c89fe77d8ed96e88f57c7"
server_key_iv: "8329b6afac2ded2a4c8e871522177f3e"
server_key_version: "encryptionkey@0"
I20260810 14:44:45.283269 15370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:44:45.285729 15391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:45.286998 15370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:44:45.287133 15370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "b5a005977fdf4982ab8f348919538250"
format_stamp: "Formatted at 2026-08-10 14:44:45 on dist-test-slave-rmhg"
server_key: "24295571305c89fe77d8ed96e88f57c7"
server_key_iv: "8329b6afac2ded2a4c8e871522177f3e"
server_key_version: "encryptionkey@0"
I20260810 14:44:45.287245 15370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:45.291996 15370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:45.294915 15370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:45.295083 15370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:45.302052 15370 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40039
I20260810 14:44:45.302088 15461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40039 every 8 connection(s)
I20260810 14:44:45.303112 15370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:45.306022 15462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:45.312013 15462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250: Bootstrap starting.
I20260810 14:44:45.312561 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 15370
I20260810 14:44:45.312705 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:45.312973 15696 external_mini_cluster.cc:1428] Setting key 0e037f5b1a76a3d45df2c7bcc2a57ded
I20260810 14:44:45.314844 15462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:45.315589 15462 log.cc:824] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:45.317903 15462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250: No bootstrap required, opened a new log
I20260810 14:44:45.318563 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:45.314443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45671 (local address 127.0.0.1:40039)
0810 14:44:45.314904 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:45.314920 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:45.314960 (+ 40us) server_negotiation.cc:407] Connection header received
0810 14:44:45.315662 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:45.315681 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:45.316036 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:45.316423 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:45.317276 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:45.317281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:45.317298 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:45.317310 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:45.317373 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:45.317394 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:45.317520 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:45.317616 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 14:44:45.317775 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":128,"threads_started":1}
I20260810 14:44:45.321417 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:44:45.321697 15462 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a005977fdf4982ab8f348919538250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } }
I20260810 14:44:45.322006 15462 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:45.322067 15462 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a005977fdf4982ab8f348919538250, State: Initialized, Role: FOLLOWER
I20260810 14:44:45.322556 15462 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a005977fdf4982ab8f348919538250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } }
I20260810 14:44:45.322734 15462 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:45.322809 15462 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:45.322921 15462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:45.324198 15462 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a005977fdf4982ab8f348919538250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } }
I20260810 14:44:45.324623 15462 leader_election.cc:302] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5a005977fdf4982ab8f348919538250; no voters:
I20260810 14:44:45.325047 15462 leader_election.cc:288] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:45.325116 15482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:45.325393 15482 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [term 1 LEADER]: Becoming Leader. State: Replica: b5a005977fdf4982ab8f348919538250, State: Running, Role: LEADER
I20260810 14:44:45.325722 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:45.323875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54751 (local address 127.0.0.1:40039)
0810 14:44:45.323999 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:45.324004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:45.324185 (+ 181us) server_negotiation.cc:407] Connection header received
0810 14:44:45.324353 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:45.324356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:45.324396 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:45.324476 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:45.325255 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:45.325259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:45.325261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:45.325275 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:45.325305 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:45.325320 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:45.325379 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:45.325565 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 14:44:45.325607 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 14:44:45.325769 15482 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a005977fdf4982ab8f348919538250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } }
I20260810 14:44:45.326570 15462 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:45.327077 15482 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5a005977fdf4982ab8f348919538250. Latest consensus state: current_term: 1 leader_uuid: "b5a005977fdf4982ab8f348919538250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a005977fdf4982ab8f348919538250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } } }
I20260810 14:44:45.327272 15482 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:45.327678 15484 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5a005977fdf4982ab8f348919538250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a005977fdf4982ab8f348919538250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } } }
I20260810 14:44:45.327773 15484 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:45.328066 15489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:45.331462 15489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:45.337920 15489 catalog_manager.cc:1347] Generated new cluster ID: 66e6457ab7764df9960ea949b140571f
I20260810 14:44:45.338006 15489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:45.350055 15489 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:45.350419 15489 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:45.359928 15489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5a005977fdf4982ab8f348919538250: Generated new TSK 0
I20260810 14:44:45.360910 15489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:45.370342 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 15370
2026-08-10T14:44:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1529 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:45 dist-test-slave-rmhg krb5kdc[15511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:45 dist-test-slave-rmhg krb5kdc[15511](info): set up 2 sockets
Aug 10 14:44:45 dist-test-slave-rmhg krb5kdc[15511](info): commencing operation
krb5kdc: starting...
W20260810 14:44:46.517856 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:46 dist-test-slave-rmhg krb5kdc[15511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:44:46.644410 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46579
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:46.761874 15531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:46.762180 15531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:46.762225 15531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:46.766498 15531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:46.766587 15531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:46.766605 15531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:46.766621 15531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:46.766635 15531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:46.772337 15531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:46.773561 15531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:46.774938 15531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:44:46.784559 15531 server_base.cc:1034] running on GCE node
W20260810 14:44:46.784504 15541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:46.784504 15539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:46.784508 15537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:46.785426 15531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:46.786650 15531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:46.787844 15531 hybrid_clock.cc:648] HybridClock initialized: now 1786373086787823 us; error 38 us; skew 500 ppm
Aug 10 14:44:46 dist-test-slave-rmhg krb5kdc[15511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:46.790859 15531 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:44:46.791882 15531 webserver.cc:460] Webserver started at http://127.0.0.1:41959/ using document root <none> and password file <none>
I20260810 14:44:46.792433 15531 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:46.792495 15531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:46.792685 15531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:46.794891 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "80bbaed8fd08445eaeeb061a1b8de136"
format_stamp: "Formatted at 2026-08-10 14:44:46 on dist-test-slave-rmhg"
server_key: "d776e64e53807b028218389a128a949b"
server_key_iv: "3f1d636abc34a53cefd6c6582e815580"
server_key_version: "encryptionkey@0"
I20260810 14:44:46.795440 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "80bbaed8fd08445eaeeb061a1b8de136"
format_stamp: "Formatted at 2026-08-10 14:44:46 on dist-test-slave-rmhg"
server_key: "d776e64e53807b028218389a128a949b"
server_key_iv: "3f1d636abc34a53cefd6c6582e815580"
server_key_version: "encryptionkey@0"
I20260810 14:44:46.799783 15531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:44:46.802479 15549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:46.803813 15531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:44:46.803949 15531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "80bbaed8fd08445eaeeb061a1b8de136"
format_stamp: "Formatted at 2026-08-10 14:44:46 on dist-test-slave-rmhg"
server_key: "d776e64e53807b028218389a128a949b"
server_key_iv: "3f1d636abc34a53cefd6c6582e815580"
server_key_version: "encryptionkey@0"
I20260810 14:44:46.804054 15531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:46.834805 15531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:46.838101 15531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:46.838294 15531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:46.845721 15531 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46579
I20260810 14:44:46.845738 15626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46579 every 8 connection(s)
I20260810 14:44:46.846707 15531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:46.849670 15627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:46.852223 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 15531
I20260810 14:44:46.852384 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:46.852698 15696 external_mini_cluster.cc:1428] Setting key fd5ccc6479aa5128a83212b038a0beb1
I20260810 14:44:46.856196 15627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136: Bootstrap starting.
I20260810 14:44:46.858644 15627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:46.859464 15627 log.cc:824] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:46.861632 15627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136: No bootstrap required, opened a new log
Aug 10 14:44:46 dist-test-slave-rmhg krb5kdc[15511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 14:44:46.865079 15627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bbaed8fd08445eaeeb061a1b8de136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46579 } }
I20260810 14:44:46.865406 15627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:46.865500 15627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bbaed8fd08445eaeeb061a1b8de136, State: Initialized, Role: FOLLOWER
I20260810 14:44:46.866082 15627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bbaed8fd08445eaeeb061a1b8de136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46579 } }
I20260810 14:44:46.866262 15627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:46.866338 15627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:46.866454 15627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:46.867748 15627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bbaed8fd08445eaeeb061a1b8de136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46579 } }
I20260810 14:44:46.868162 15627 leader_election.cc:302] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80bbaed8fd08445eaeeb061a1b8de136; no voters:
I20260810 14:44:46.868251 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:46.854280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59983 (local address 127.0.0.1:46579)
0810 14:44:46.854748 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:46.854765 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:46.854802 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:44:46.855466 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:46.855484 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:46.855854 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:46.856203 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:46.857036 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:46.858131 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:46.859086 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:46.859616 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:46.859782 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:46.862447 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:46.862452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:46.862468 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:46.862499 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:46.864651 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:46.865232 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:46.865238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:46.865243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:46.865366 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:46.865696 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:46.865700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:46.865702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:46.866286 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:46.866496 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:46.866796 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 14:44:46.867140 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":136,"threads_started":1}
I20260810 14:44:46.868535 15627 leader_election.cc:288] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:46.868645 15634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:46.868919 15634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [term 1 LEADER]: Becoming Leader. State: Replica: 80bbaed8fd08445eaeeb061a1b8de136, State: Running, Role: LEADER
I20260810 14:44:46.869372 15634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bbaed8fd08445eaeeb061a1b8de136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46579 } }
I20260810 14:44:46.870018 15627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:46.871016 15636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80bbaed8fd08445eaeeb061a1b8de136. Latest consensus state: current_term: 1 leader_uuid: "80bbaed8fd08445eaeeb061a1b8de136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bbaed8fd08445eaeeb061a1b8de136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46579 } } }
I20260810 14:44:46.871156 15636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:46.871163 15635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80bbaed8fd08445eaeeb061a1b8de136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bbaed8fd08445eaeeb061a1b8de136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46579 } } }
I20260810 14:44:46.871447 15635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:46.871928 15648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:46.872810 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:44:46.875607 15648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:46.882028 15648 catalog_manager.cc:1347] Generated new cluster ID: 0f3f4dbf3acb42849fc580d6d30dad03
I20260810 14:44:46.882159 15648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:46.884248 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:46.875260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49285 (local address 127.0.0.1:46579)
0810 14:44:46.875379 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:46.875383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:46.875596 (+ 213us) server_negotiation.cc:407] Connection header received
0810 14:44:46.875771 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:46.875775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:46.875833 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:46.875936 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:46.876786 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:46.877702 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:46.878827 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:46.879227 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:46.879339 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:46.880798 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:46.880801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:46.880803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:46.880828 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:46.882474 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:46.882972 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:46.882976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:46.882978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:46.883066 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:46.883364 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:46.883367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:46.883369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:46.883563 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:46.883653 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:46.883973 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 14:44:46.884089 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:44:46.901353 15648 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:46.902391 15648 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:46.909478 15648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80bbaed8fd08445eaeeb061a1b8de136: Generated new TSK 0
I20260810 14:44:46.910215 15648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:46.920305 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 15531
2026-08-10T14:44:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1553 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:46 dist-test-slave-rmhg krb5kdc[15675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:46 dist-test-slave-rmhg krb5kdc[15675](info): set up 2 sockets
Aug 10 14:44:46 dist-test-slave-rmhg krb5kdc[15675](info): commencing operation
krb5kdc: starting...
W20260810 14:44:48.062572 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:48 dist-test-slave-rmhg krb5kdc[15675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:44:48.189312 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36545
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:48.306497 15693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:48.306792 15693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:48.306838 15693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:48.311015 15693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:48.311105 15693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:48.311123 15693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:48.311141 15693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:48.311154 15693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:48.317020 15693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:48.318291 15693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:48.319370 15693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:48.328333 15710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:48.328372 15693 server_base.cc:1034] running on GCE node
W20260810 14:44:48.328327 15706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:48.328327 15714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:48.329294 15693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:48.330518 15693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:48.331692 15693 hybrid_clock.cc:648] HybridClock initialized: now 1786373088331677 us; error 33 us; skew 500 ppm
Aug 10 14:44:48 dist-test-slave-rmhg krb5kdc[15675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:48.334613 15693 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:44:48.335608 15693 webserver.cc:460] Webserver started at http://127.0.0.1:36125/ using document root <none> and password file <none>
I20260810 14:44:48.336186 15693 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:48.336249 15693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:48.336431 15693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:48.338619 15693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "5dbb524dcb124d57b7431652b3da53a6"
format_stamp: "Formatted at 2026-08-10 14:44:48 on dist-test-slave-rmhg"
server_key: "ab4c292c9997b450c6e125ed3b22fb40"
server_key_iv: "dce4934499b601bade73be30a1896127"
server_key_version: "encryptionkey@0"
I20260810 14:44:48.339178 15693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "5dbb524dcb124d57b7431652b3da53a6"
format_stamp: "Formatted at 2026-08-10 14:44:48 on dist-test-slave-rmhg"
server_key: "ab4c292c9997b450c6e125ed3b22fb40"
server_key_iv: "dce4934499b601bade73be30a1896127"
server_key_version: "encryptionkey@0"
I20260810 14:44:48.343086 15693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:44:48.345659 15721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:48.347018 15693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:44:48.347147 15693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "5dbb524dcb124d57b7431652b3da53a6"
format_stamp: "Formatted at 2026-08-10 14:44:48 on dist-test-slave-rmhg"
server_key: "ab4c292c9997b450c6e125ed3b22fb40"
server_key_iv: "dce4934499b601bade73be30a1896127"
server_key_version: "encryptionkey@0"
I20260810 14:44:48.347268 15693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:48.358018 15693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:48.361287 15693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:48.361486 15693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:48.368677 15693 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36545
I20260810 14:44:48.368705 15804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36545 every 8 connection(s)
I20260810 14:44:48.369727 15693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:48.372677 15805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:48.376780 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 15693
I20260810 14:44:48.376951 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:48.377308 15696 external_mini_cluster.cc:1428] Setting key 81660306b3bd9e7aeccb0fc71108d16a
I20260810 14:44:48.379451 15805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6: Bootstrap starting.
I20260810 14:44:48.382493 15805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:48.383518 15805 log.cc:824] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:48.386142 15805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6: No bootstrap required, opened a new log
I20260810 14:44:48.388336 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:48.378875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33891 (local address 127.0.0.1:36545)
0810 14:44:48.379656 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:48.379678 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:48.379724 (+ 46us) server_negotiation.cc:407] Connection header received
0810 14:44:48.380532 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:48.380555 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:48.380971 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:48.381389 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:48.381864 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:48.383176 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:48.384596 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:48.385370 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:48.385539 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:48.386383 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:48.386389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:48.386405 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:48.386804 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:48.386864 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:48.386887 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:48.386977 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:48.387141 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 14:44:48.387366 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":170,"threads_started":1}
I20260810 14:44:48.389731 15805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbb524dcb124d57b7431652b3da53a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260810 14:44:48.390048 15805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:48.390106 15805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dbb524dcb124d57b7431652b3da53a6, State: Initialized, Role: FOLLOWER
I20260810 14:44:48.390583 15805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbb524dcb124d57b7431652b3da53a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260810 14:44:48.390728 15805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:48.390792 15805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:48.390898 15805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:48.391582 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:44:48.391995 15805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbb524dcb124d57b7431652b3da53a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260810 14:44:48.392335 15805 leader_election.cc:302] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dbb524dcb124d57b7431652b3da53a6; no voters:
I20260810 14:44:48.392642 15805 leader_election.cc:288] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:48.392733 15815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:48.392928 15815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5dbb524dcb124d57b7431652b3da53a6, State: Running, Role: LEADER
I20260810 14:44:48.393291 15815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbb524dcb124d57b7431652b3da53a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260810 14:44:48.394070 15805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:48.394573 15815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dbb524dcb124d57b7431652b3da53a6. Latest consensus state: current_term: 1 leader_uuid: "5dbb524dcb124d57b7431652b3da53a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbb524dcb124d57b7431652b3da53a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } } }
I20260810 14:44:48.394691 15815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:48.395001 15819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dbb524dcb124d57b7431652b3da53a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbb524dcb124d57b7431652b3da53a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } } }
I20260810 14:44:48.395116 15819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6 [sys.catalog]: This master's current role is: LEADER
W20260810 14:44:48.399371 15841 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:44:48.399459 15841 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:44:48.399521 15824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:48.400435 15824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:48.400446 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:48.395282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58559 (local address 127.0.0.1:36545)
0810 14:44:48.395391 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:48.395396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:48.395805 (+ 409us) server_negotiation.cc:407] Connection header received
0810 14:44:48.395967 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:48.395971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:48.396025 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:48.396108 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:48.396891 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:48.397476 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:48.398592 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:48.398947 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:48.399085 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:48.399961 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:48.399966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:48.399969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:48.399983 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:48.400025 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:48.400040 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:48.400119 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:48.400239 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 14:44:48.400302 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:44:48.405822 15824 catalog_manager.cc:1347] Generated new cluster ID: 78088031618a415fa4b433e2122fc0ad
I20260810 14:44:48.405903 15824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:48.441740 15824 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:48.442812 15824 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:48.450917 15824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dbb524dcb124d57b7431652b3da53a6: Generated new TSK 0
I20260810 14:44:48.451802 15824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:48.456849 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 15693
2026-08-10T14:44:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1536 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:48 dist-test-slave-rmhg krb5kdc[15850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:48 dist-test-slave-rmhg krb5kdc[15850](info): set up 2 sockets
Aug 10 14:44:48 dist-test-slave-rmhg krb5kdc[15850](info): commencing operation
krb5kdc: starting...
W20260810 14:44:49.611047 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:49 dist-test-slave-rmhg krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:44:49.742437 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:49.859346 15869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:49.859642 15869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:49.859714 15869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:49.863773 15869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:49.863859 15869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:49.863888 15869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:49.863909 15869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:49.863931 15869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:49.869560 15869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:49.870738 15869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:49.871793 15869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:44:49.880864 15869 server_base.cc:1034] running on GCE node
W20260810 14:44:49.880807 15876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:49.880807 15879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:49.880818 15877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:49.881733 15869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:49.882908 15869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:49.884080 15869 hybrid_clock.cc:648] HybridClock initialized: now 1786373089884055 us; error 44 us; skew 500 ppm
Aug 10 14:44:49 dist-test-slave-rmhg krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:49.886873 15869 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:44:49.887836 15869 webserver.cc:460] Webserver started at http://127.0.0.1:43349/ using document root <none> and password file <none>
I20260810 14:44:49.888412 15869 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:49.888482 15869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:49.888695 15869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:49.890919 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "59ed37bae2914a6598aa6b3526231cf8"
format_stamp: "Formatted at 2026-08-10 14:44:49 on dist-test-slave-rmhg"
server_key: "728ad8b84fc1b71f70f51a1f04f79f8c"
server_key_iv: "bbb1582dfc17dcf1eae251e3a463848d"
server_key_version: "encryptionkey@0"
I20260810 14:44:49.891494 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "59ed37bae2914a6598aa6b3526231cf8"
format_stamp: "Formatted at 2026-08-10 14:44:49 on dist-test-slave-rmhg"
server_key: "728ad8b84fc1b71f70f51a1f04f79f8c"
server_key_iv: "bbb1582dfc17dcf1eae251e3a463848d"
server_key_version: "encryptionkey@0"
I20260810 14:44:49.895417 15869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 14:44:49.898032 15890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:49.899348 15869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:44:49.899480 15869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "59ed37bae2914a6598aa6b3526231cf8"
format_stamp: "Formatted at 2026-08-10 14:44:49 on dist-test-slave-rmhg"
server_key: "728ad8b84fc1b71f70f51a1f04f79f8c"
server_key_iv: "bbb1582dfc17dcf1eae251e3a463848d"
server_key_version: "encryptionkey@0"
I20260810 14:44:49.899595 15869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:49.904501 15869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:49.907486 15869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:49.907683 15869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:49.914870 15869 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44849
I20260810 14:44:49.914901 15978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44849 every 8 connection(s)
I20260810 14:44:49.915916 15869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:49.919061 15979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:49.920011 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 15869
I20260810 14:44:49.920162 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:49.920485 15696 external_mini_cluster.cc:1428] Setting key 58a0f29265eb9d355adf30352eddb5a6
I20260810 14:44:49.926035 15979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8: Bootstrap starting.
I20260810 14:44:49.926296 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:49.922102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36329 (local address 127.0.0.1:44849)
0810 14:44:49.922557 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:49.922575 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:49.922613 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:44:49.923279 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:49.923297 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:49.923623 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:49.923933 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:49.924864 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:49.924869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:49.924883 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:49.924894 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:49.924955 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:49.924977 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:49.925100 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:49.925261 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 14:44:49.925434 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":123,"threads_started":1}
I20260810 14:44:49.928700 15979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:49.929486 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:44:49.929673 15979 log.cc:824] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:49.932341 15979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8: No bootstrap required, opened a new log
I20260810 14:44:49.933776 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:49.931976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54223 (local address 127.0.0.1:44849)
0810 14:44:49.932107 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:49.932111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:49.932284 (+ 173us) server_negotiation.cc:407] Connection header received
0810 14:44:49.932428 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:49.932431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:49.932469 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:49.932538 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:49.933346 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:49.933351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:49.933354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:49.933366 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:49.933394 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:49.933405 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:49.933481 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:49.933614 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 14:44:49.933654 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 14:44:49.936470 15979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ed37bae2914a6598aa6b3526231cf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44849 } }
I20260810 14:44:49.936775 15979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:49.936837 15979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ed37bae2914a6598aa6b3526231cf8, State: Initialized, Role: FOLLOWER
I20260810 14:44:49.937321 15979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ed37bae2914a6598aa6b3526231cf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44849 } }
I20260810 14:44:49.937458 15979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:49.937523 15979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:49.937621 15979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:49.938596 15979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ed37bae2914a6598aa6b3526231cf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44849 } }
I20260810 14:44:49.938947 15979 leader_election.cc:302] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59ed37bae2914a6598aa6b3526231cf8; no voters:
I20260810 14:44:49.939270 15979 leader_election.cc:288] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:49.939373 15994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:49.939589 15994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [term 1 LEADER]: Becoming Leader. State: Replica: 59ed37bae2914a6598aa6b3526231cf8, State: Running, Role: LEADER
I20260810 14:44:49.939932 15994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ed37bae2914a6598aa6b3526231cf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44849 } }
I20260810 14:44:49.940508 15979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:49.941862 15995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59ed37bae2914a6598aa6b3526231cf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ed37bae2914a6598aa6b3526231cf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44849 } } }
I20260810 14:44:49.941998 15995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:49.942358 15996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59ed37bae2914a6598aa6b3526231cf8. Latest consensus state: current_term: 1 leader_uuid: "59ed37bae2914a6598aa6b3526231cf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ed37bae2914a6598aa6b3526231cf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44849 } } }
I20260810 14:44:49.942441 16007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:49.942456 15996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:49.945724 16007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:49.951812 16007 catalog_manager.cc:1347] Generated new cluster ID: fa154b4489664b918948c9b1b2218a90
I20260810 14:44:49.951906 16007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:49.962657 16007 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:49.962996 16007 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:49.968952 16007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59ed37bae2914a6598aa6b3526231cf8: Generated new TSK 0
I20260810 14:44:49.969700 16007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:49.979322 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 15869
2026-08-10T14:44:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1525 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:50 dist-test-slave-rmhg krb5kdc[16024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:50 dist-test-slave-rmhg krb5kdc[16024](info): set up 2 sockets
Aug 10 14:44:50 dist-test-slave-rmhg krb5kdc[16024](info): commencing operation
krb5kdc: starting...
W20260810 14:44:51.136605 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:51 dist-test-slave-rmhg krb5kdc[16024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:44:51.264652 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44841
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:51.380227 16044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:51.380532 16044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:51.380573 16044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:51.384687 16044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:51.384770 16044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:51.384788 16044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:51.384804 16044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:51.384828 16044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:51.390372 16044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:51.391494 16044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:51.392501 16044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:51.402467 16051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:51.402516 16052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:51.402515 16044 server_base.cc:1034] running on GCE node
W20260810 14:44:51.402451 16055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:51.403314 16044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:51.404501 16044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:51.405666 16044 hybrid_clock.cc:648] HybridClock initialized: now 1786373091405636 us; error 48 us; skew 500 ppm
Aug 10 14:44:51 dist-test-slave-rmhg krb5kdc[16024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:51.417513 16044 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:44:51.418613 16044 webserver.cc:460] Webserver started at http://127.0.0.1:33389/ using document root <none> and password file <none>
I20260810 14:44:51.419204 16044 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:51.419277 16044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:51.419498 16044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:51.421731 16044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "7906141f82c743a092013f500f1c61b8"
format_stamp: "Formatted at 2026-08-10 14:44:51 on dist-test-slave-rmhg"
server_key: "cc73f32674439ee7bddd68b341d93374"
server_key_iv: "acd45a54cc468259123011483d020f3b"
server_key_version: "encryptionkey@0"
I20260810 14:44:51.422346 16044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "7906141f82c743a092013f500f1c61b8"
format_stamp: "Formatted at 2026-08-10 14:44:51 on dist-test-slave-rmhg"
server_key: "cc73f32674439ee7bddd68b341d93374"
server_key_iv: "acd45a54cc468259123011483d020f3b"
server_key_version: "encryptionkey@0"
I20260810 14:44:51.426553 16044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:44:51.429309 16063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:51.430701 16044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:44:51.430861 16044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "7906141f82c743a092013f500f1c61b8"
format_stamp: "Formatted at 2026-08-10 14:44:51 on dist-test-slave-rmhg"
server_key: "cc73f32674439ee7bddd68b341d93374"
server_key_iv: "acd45a54cc468259123011483d020f3b"
server_key_version: "encryptionkey@0"
I20260810 14:44:51.430984 16044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:51.455665 16044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:51.458854 16044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:51.459064 16044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:51.466580 16044 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44841
I20260810 14:44:51.466594 16157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44841 every 8 connection(s)
I20260810 14:44:51.467595 16044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:51.470494 16158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:51.472205 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 16044
I20260810 14:44:51.472357 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:51.472640 15696 external_mini_cluster.cc:1428] Setting key e659d90c5e69b4cd97f742996bf3195e
I20260810 14:44:51.477142 16158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8: Bootstrap starting.
I20260810 14:44:51.479749 16158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:51.480638 16158 log.cc:824] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8: Log is configured to *not* fsync() on all Append() calls
Aug 10 14:44:51 dist-test-slave-rmhg krb5kdc[16024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 14:44:51.482822 16158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8: No bootstrap required, opened a new log
I20260810 14:44:51.487525 16158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7906141f82c743a092013f500f1c61b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44841 } }
I20260810 14:44:51.487780 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:51.474182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:57371 (local address 127.0.0.1:44841)
0810 14:44:51.474623 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:51.474640 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:51.474678 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:44:51.475319 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:51.475337 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:51.475700 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:51.475990 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:51.476899 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:51.477970 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:51.479138 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:51.479622 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:51.479745 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:51.482497 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:51.482504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:51.482520 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:51.482553 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:51.484441 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:51.484981 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:51.484988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:51.484995 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:51.485215 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:51.485513 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:51.485516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:51.485518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:51.486055 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:51.486300 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:51.486511 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 14:44:51.486761 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20260810 14:44:51.487828 16158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:51.487883 16158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7906141f82c743a092013f500f1c61b8, State: Initialized, Role: FOLLOWER
I20260810 14:44:51.488371 16158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7906141f82c743a092013f500f1c61b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44841 } }
I20260810 14:44:51.488533 16158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:51.488607 16158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:51.488703 16158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:51.489758 16158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7906141f82c743a092013f500f1c61b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44841 } }
I20260810 14:44:51.490115 16158 leader_election.cc:302] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7906141f82c743a092013f500f1c61b8; no voters:
I20260810 14:44:51.490476 16158 leader_election.cc:288] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:51.490578 16165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:51.490789 16165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7906141f82c743a092013f500f1c61b8, State: Running, Role: LEADER
I20260810 14:44:51.490937 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:44:51.491118 16165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7906141f82c743a092013f500f1c61b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44841 } }
I20260810 14:44:51.491698 16158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:51.492406 16165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7906141f82c743a092013f500f1c61b8. Latest consensus state: current_term: 1 leader_uuid: "7906141f82c743a092013f500f1c61b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7906141f82c743a092013f500f1c61b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44841 } } }
I20260810 14:44:51.492529 16165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:51.492911 16167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7906141f82c743a092013f500f1c61b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7906141f82c743a092013f500f1c61b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44841 } } }
I20260810 14:44:51.493024 16167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8 [sys.catalog]: This master's current role is: LEADER
W20260810 14:44:51.496670 16201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:44:51.496768 16201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:44:51.496891 16185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:51.497851 16185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:51.503199 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:51.493299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:37557 (local address 127.0.0.1:44841)
0810 14:44:51.493416 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:51.493421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:51.493603 (+ 182us) server_negotiation.cc:407] Connection header received
0810 14:44:51.493741 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:51.493745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:51.493801 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:51.493885 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:51.494825 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:51.495573 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:51.496591 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:51.496839 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:51.496941 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:51.498644 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:51.498648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:51.498651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:51.498678 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:51.501079 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:51.501739 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:51.501743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:51.501744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:51.501843 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:51.502191 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:51.502195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:51.502197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:51.502433 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:51.502581 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:51.502943 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 14:44:51.503046 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:44:51.504622 16185 catalog_manager.cc:1347] Generated new cluster ID: 3a0c4effb6df4107ae5a98d746d0c2ca
I20260810 14:44:51.504681 16185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:51.522635 16185 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:51.523701 16185 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:51.529765 16185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7906141f82c743a092013f500f1c61b8: Generated new TSK 0
I20260810 14:44:51.530560 16185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:51.539606 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 16044
W20260810 14:44:51.553473 16021 connection.cc:535] client connection to 127.0.0.1:44841 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44841): Connection reset by peer (error 104)
2026-08-10T14:44:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1554 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:51 dist-test-slave-rmhg krb5kdc[16209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:51 dist-test-slave-rmhg krb5kdc[16209](info): set up 2 sockets
Aug 10 14:44:51 dist-test-slave-rmhg krb5kdc[16209](info): commencing operation
krb5kdc: starting...
W20260810 14:44:52.697888 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:52 dist-test-slave-rmhg krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:44:52.834324 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:52.950645 16228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:52.951004 16228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:52.951053 16228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:52.955116 16228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:52.955186 16228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:52.955205 16228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:52.955221 16228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:52.955235 16228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:52.960728 16228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:52.961962 16228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:52.962994 16228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:52.971704 16243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:52.971866 16228 server_base.cc:1034] running on GCE node
W20260810 14:44:52.971719 16245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:52.971711 16237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:52.972586 16228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:52.973793 16228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:52.974959 16228 hybrid_clock.cc:648] HybridClock initialized: now 1786373092974946 us; error 29 us; skew 500 ppm
Aug 10 14:44:52 dist-test-slave-rmhg krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:52.977733 16228 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:44:52.978670 16228 webserver.cc:460] Webserver started at http://127.0.0.1:37893/ using document root <none> and password file <none>
I20260810 14:44:52.979246 16228 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:52.979317 16228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:52.979548 16228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:52.981724 16228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "b1196c55098b4db09c681bd62631c519"
format_stamp: "Formatted at 2026-08-10 14:44:52 on dist-test-slave-rmhg"
server_key: "267a190ddfe20c76ccf2c64d9169a956"
server_key_iv: "f381750338067a7c48945ec6ab5d0db2"
server_key_version: "encryptionkey@0"
I20260810 14:44:52.982288 16228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "b1196c55098b4db09c681bd62631c519"
format_stamp: "Formatted at 2026-08-10 14:44:52 on dist-test-slave-rmhg"
server_key: "267a190ddfe20c76ccf2c64d9169a956"
server_key_iv: "f381750338067a7c48945ec6ab5d0db2"
server_key_version: "encryptionkey@0"
I20260810 14:44:52.986137 16228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:44:52.988740 16254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:52.990056 16228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:44:52.990195 16228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "b1196c55098b4db09c681bd62631c519"
format_stamp: "Formatted at 2026-08-10 14:44:52 on dist-test-slave-rmhg"
server_key: "267a190ddfe20c76ccf2c64d9169a956"
server_key_iv: "f381750338067a7c48945ec6ab5d0db2"
server_key_version: "encryptionkey@0"
I20260810 14:44:52.990304 16228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:53.005918 16228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:53.009239 16228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:53.009443 16228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:53.016592 16228 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45337
I20260810 14:44:53.016633 16334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45337 every 8 connection(s)
I20260810 14:44:53.017670 16228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:53.020601 16336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:53.022171 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 16228
I20260810 14:44:53.022315 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:53.022574 15696 external_mini_cluster.cc:1428] Setting key 0c503327f5c8265ce6d8ec67bb43837c
I20260810 14:44:53.027402 16336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519: Bootstrap starting.
I20260810 14:44:53.030159 16336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:53.031085 16336 log.cc:824] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:53.032729 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:53.024028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:54579 (local address 127.0.0.1:45337)
0810 14:44:53.024487 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:53.024508 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:53.024563 (+ 55us) server_negotiation.cc:407] Connection header received
0810 14:44:53.025413 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:53.025438 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:53.025917 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:53.026305 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:53.026746 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:53.028200 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:53.029089 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:53.029811 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:53.029983 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:53.030815 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:53.030822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:53.030839 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:53.031352 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:53.031427 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:53.031461 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:53.031607 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:53.031770 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 14:44:53.031913 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":120,"threads_started":1}
I20260810 14:44:53.033301 16336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519: No bootstrap required, opened a new log
I20260810 14:44:53.035706 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:44:53.037497 16336 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1196c55098b4db09c681bd62631c519" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } }
I20260810 14:44:53.037823 16336 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:53.037905 16336 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1196c55098b4db09c681bd62631c519, State: Initialized, Role: FOLLOWER
I20260810 14:44:53.038398 16336 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1196c55098b4db09c681bd62631c519" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } }
I20260810 14:44:53.038538 16336 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:53.038609 16336 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:53.038717 16336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:53.039686 16336 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1196c55098b4db09c681bd62631c519" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } }
I20260810 14:44:53.040035 16336 leader_election.cc:302] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1196c55098b4db09c681bd62631c519; no voters:
I20260810 14:44:53.040365 16336 leader_election.cc:288] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:53.040453 16350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:53.040635 16350 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [term 1 LEADER]: Becoming Leader. State: Replica: b1196c55098b4db09c681bd62631c519, State: Running, Role: LEADER
I20260810 14:44:53.040966 16350 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1196c55098b4db09c681bd62631c519" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } }
I20260810 14:44:53.041817 16336 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:53.042644 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:53.038167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:39519 (local address 127.0.0.1:45337)
0810 14:44:53.038280 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:53.038284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:53.038498 (+ 214us) server_negotiation.cc:407] Connection header received
0810 14:44:53.038629 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:53.038632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:53.038683 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:53.038754 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:53.039017 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:53.039539 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:53.040509 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:53.040843 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:53.040975 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:53.041892 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:53.041896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:53.041899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:53.041914 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:53.041945 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:53.041958 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:53.042044 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:53.042466 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 14:44:53.042504 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 14:44:53.042706 16351 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1196c55098b4db09c681bd62631c519" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1196c55098b4db09c681bd62631c519" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } } }
I20260810 14:44:53.043223 16351 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:53.043481 16350 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1196c55098b4db09c681bd62631c519. Latest consensus state: current_term: 1 leader_uuid: "b1196c55098b4db09c681bd62631c519" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1196c55098b4db09c681bd62631c519" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } } }
I20260810 14:44:53.044112 16350 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519 [sys.catalog]: This master's current role is: LEADER
W20260810 14:44:53.046870 16377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:44:53.046952 16377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:44:53.047031 16366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:53.047761 16366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:53.054069 16366 catalog_manager.cc:1347] Generated new cluster ID: e573d7962e7941e691a0c1f1bdc1fbbc
I20260810 14:44:53.054149 16366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:53.066888 16366 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:53.067915 16366 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:53.079238 16366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1196c55098b4db09c681bd62631c519: Generated new TSK 0
I20260810 14:44:53.080010 16366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:53.093783 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 16228
W20260810 14:44:53.107352 16207 connection.cc:535] client connection to 127.0.0.1:45337 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45337): Connection reset by peer (error 104)
2026-08-10T14:44:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1553 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:53 dist-test-slave-rmhg krb5kdc[16389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:53 dist-test-slave-rmhg krb5kdc[16389](info): set up 2 sockets
Aug 10 14:44:53 dist-test-slave-rmhg krb5kdc[16389](info): commencing operation
krb5kdc: starting...
W20260810 14:44:54.244262 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:54 dist-test-slave-rmhg krb5kdc[16389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:44:54.376220 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38025
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:54.495677 16408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:54.495972 16408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:54.496016 16408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:54.500180 16408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:54.500265 16408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:54.500284 16408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:54.500300 16408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:54.500313 16408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:54.506074 16408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:54.507248 16408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:54.508335 16408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:54.517397 16419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:54.517436 16416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:54.517397 16417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:54.517709 16408 server_base.cc:1034] running on GCE node
I20260810 14:44:54.518425 16408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:54.519757 16408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:54.520931 16408 hybrid_clock.cc:648] HybridClock initialized: now 1786373094520908 us; error 39 us; skew 500 ppm
Aug 10 14:44:54 dist-test-slave-rmhg krb5kdc[16389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:54.523783 16408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:44:54.524730 16408 webserver.cc:460] Webserver started at http://127.0.0.1:33623/ using document root <none> and password file <none>
I20260810 14:44:54.525327 16408 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:54.525399 16408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:54.525624 16408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:54.527740 16408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "696451a4f103451dabf246987daccd64"
format_stamp: "Formatted at 2026-08-10 14:44:54 on dist-test-slave-rmhg"
server_key: "25a46751a9d011423af9a3ebdc0ccc7b"
server_key_iv: "8f3f79dc9184efc18440a055332dcdc4"
server_key_version: "encryptionkey@0"
I20260810 14:44:54.528322 16408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "696451a4f103451dabf246987daccd64"
format_stamp: "Formatted at 2026-08-10 14:44:54 on dist-test-slave-rmhg"
server_key: "25a46751a9d011423af9a3ebdc0ccc7b"
server_key_iv: "8f3f79dc9184efc18440a055332dcdc4"
server_key_version: "encryptionkey@0"
I20260810 14:44:54.532321 16408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 14:44:54.535028 16431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:54.536358 16408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:44:54.536494 16408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "696451a4f103451dabf246987daccd64"
format_stamp: "Formatted at 2026-08-10 14:44:54 on dist-test-slave-rmhg"
server_key: "25a46751a9d011423af9a3ebdc0ccc7b"
server_key_iv: "8f3f79dc9184efc18440a055332dcdc4"
server_key_version: "encryptionkey@0"
I20260810 14:44:54.536625 16408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:54.541435 16408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:54.544579 16408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:54.544761 16408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:54.551837 16408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38025
I20260810 14:44:54.551849 16508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38025 every 8 connection(s)
I20260810 14:44:54.552842 16408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:54.554387 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 16408
I20260810 14:44:54.554530 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:54.554819 15696 external_mini_cluster.cc:1428] Setting key 0f8e4d7b83fa3b6810d389c1f626e651
I20260810 14:44:54.556299 16510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:54.560516 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:54.556217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:60605 (local address 127.0.0.1:38025)
0810 14:44:54.556625 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:54.556649 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:54.556695 (+ 46us) server_negotiation.cc:407] Connection header received
0810 14:44:54.557579 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:54.557604 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:54.557978 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:54.558363 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:54.559073 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:54.559079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:54.559095 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:54.559106 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:54.559168 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:54.559194 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:54.559341 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:54.559411 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 14:44:54.559575 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":114,"threads_started":1}
I20260810 14:44:54.563329 16510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64: Bootstrap starting.
I20260810 14:44:54.563377 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:44:54.566241 16510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:54.567222 16510 log.cc:824] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:54.567739 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:54.565856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:54157 (local address 127.0.0.1:38025)
0810 14:44:54.565997 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:54.566001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:54.566190 (+ 189us) server_negotiation.cc:407] Connection header received
0810 14:44:54.566322 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:54.566325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:54.566365 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:54.566436 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:54.567170 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:54.567173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:54.567175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:54.567186 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:54.567215 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:54.567226 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:54.567316 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:54.567577 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 14:44:54.567622 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:44:54.569624 16510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64: No bootstrap required, opened a new log
I20260810 14:44:54.573549 16510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696451a4f103451dabf246987daccd64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38025 } }
I20260810 14:44:54.573829 16510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:54.573866 16510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696451a4f103451dabf246987daccd64, State: Initialized, Role: FOLLOWER
I20260810 14:44:54.574301 16510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696451a4f103451dabf246987daccd64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38025 } }
I20260810 14:44:54.574435 16510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:54.574492 16510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:54.574577 16510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:54.575541 16510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696451a4f103451dabf246987daccd64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38025 } }
I20260810 14:44:54.575874 16510 leader_election.cc:302] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 696451a4f103451dabf246987daccd64; no voters:
I20260810 14:44:54.576164 16510 leader_election.cc:288] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:54.576339 16525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:54.576606 16525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [term 1 LEADER]: Becoming Leader. State: Replica: 696451a4f103451dabf246987daccd64, State: Running, Role: LEADER
I20260810 14:44:54.576973 16525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696451a4f103451dabf246987daccd64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38025 } }
I20260810 14:44:54.577461 16510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:54.578663 16529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 696451a4f103451dabf246987daccd64. Latest consensus state: current_term: 1 leader_uuid: "696451a4f103451dabf246987daccd64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696451a4f103451dabf246987daccd64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38025 } } }
I20260810 14:44:54.578799 16529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:54.578943 16527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "696451a4f103451dabf246987daccd64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696451a4f103451dabf246987daccd64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38025 } } }
I20260810 14:44:54.579048 16527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:54.579442 16540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:54.582320 16540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:54.588021 16540 catalog_manager.cc:1347] Generated new cluster ID: dc5d39f810874f6a8807118a0030c378
I20260810 14:44:54.588107 16540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:54.601854 16540 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:54.602253 16540 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:54.608798 16540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 696451a4f103451dabf246987daccd64: Generated new TSK 0
I20260810 14:44:54.609802 16540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:54.612943 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 16408
2026-08-10T14:44:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1521 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:54 dist-test-slave-rmhg krb5kdc[16555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:54 dist-test-slave-rmhg krb5kdc[16555](info): set up 2 sockets
Aug 10 14:44:54 dist-test-slave-rmhg krb5kdc[16555](info): commencing operation
krb5kdc: starting...
W20260810 14:44:55.766769 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:55 dist-test-slave-rmhg krb5kdc[16555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:44:55.897934 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:56.013764 16579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:56.014038 16579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:56.014081 16579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:56.018102 16579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:56.018168 16579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:56.018186 16579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:56.018203 16579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:56.018216 16579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:56.023542 16579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37347
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:56.024596 16579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:56.025733 16579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:56.034030 16585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:56.034183 16579 server_base.cc:1034] running on GCE node
W20260810 14:44:56.034052 16590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:56.034049 16584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:56.034852 16579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:56.035908 16579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:56.037072 16579 hybrid_clock.cc:648] HybridClock initialized: now 1786373096037056 us; error 28 us; skew 500 ppm
Aug 10 14:44:56 dist-test-slave-rmhg krb5kdc[16555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:56.039740 16579 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:44:56.040616 16579 webserver.cc:460] Webserver started at http://127.0.0.1:38975/ using document root <none> and password file <none>
I20260810 14:44:56.041145 16579 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:56.041266 16579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:56.041477 16579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:56.043610 16579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "8f75bc74c8014076b53072c7616bf1f9"
format_stamp: "Formatted at 2026-08-10 14:44:56 on dist-test-slave-rmhg"
server_key: "b918605683cee106529f4967112c6562"
server_key_iv: "70e7e0dae9f21fc6dc449941cef33d12"
server_key_version: "encryptionkey@0"
I20260810 14:44:56.044134 16579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "8f75bc74c8014076b53072c7616bf1f9"
format_stamp: "Formatted at 2026-08-10 14:44:56 on dist-test-slave-rmhg"
server_key: "b918605683cee106529f4967112c6562"
server_key_iv: "70e7e0dae9f21fc6dc449941cef33d12"
server_key_version: "encryptionkey@0"
I20260810 14:44:56.047842 16579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 14:44:56.050235 16600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:56.051473 16579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:44:56.051600 16579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "8f75bc74c8014076b53072c7616bf1f9"
format_stamp: "Formatted at 2026-08-10 14:44:56 on dist-test-slave-rmhg"
server_key: "b918605683cee106529f4967112c6562"
server_key_iv: "70e7e0dae9f21fc6dc449941cef33d12"
server_key_version: "encryptionkey@0"
I20260810 14:44:56.051698 16579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:56.070127 16579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:56.073479 16579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:56.073678 16579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:56.080960 16579 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37347
I20260810 14:44:56.080989 16686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37347 every 8 connection(s)
I20260810 14:44:56.082217 16579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:56.085141 16688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:56.085599 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 16579
I20260810 14:44:56.085733 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:56.086006 15696 external_mini_cluster.cc:1428] Setting key 93324a7ca9e4cb2c78b5634d3b064f48
I20260810 14:44:56.092185 16688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9: Bootstrap starting.
I20260810 14:44:56.094625 16688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9: Neither blocks nor log segments found. Creating new log.
Aug 10 14:44:56 dist-test-slave-rmhg krb5kdc[16555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 14:44:56.095572 16688 log.cc:824] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:56.097827 16688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9: No bootstrap required, opened a new log
I20260810 14:44:56.100473 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:56.087499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:43733 (local address 127.0.0.1:37347)
0810 14:44:56.087994 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:56.088012 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:56.088051 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:44:56.088754 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:56.088772 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:56.089141 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:56.089543 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:56.090427 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:56.091437 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:56.092305 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:56.092836 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:56.092987 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:56.095750 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:56.095758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:56.095777 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:56.095813 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:56.097623 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:56.098135 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:56.098140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:56.098145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:56.098273 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:56.098592 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:56.098594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:56.098595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:56.099005 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:56.099220 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:56.099520 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 14:44:56.099733 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":144,"threads_started":1}
I20260810 14:44:56.101511 16688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f75bc74c8014076b53072c7616bf1f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } }
I20260810 14:44:56.101747 16688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:56.101805 16688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f75bc74c8014076b53072c7616bf1f9, State: Initialized, Role: FOLLOWER
I20260810 14:44:56.102250 16688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f75bc74c8014076b53072c7616bf1f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } }
I20260810 14:44:56.102484 16688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:56.102556 16688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:56.102696 16688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:56.103430 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:44:56.103749 16688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f75bc74c8014076b53072c7616bf1f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } }
I20260810 14:44:56.104087 16688 leader_election.cc:302] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f75bc74c8014076b53072c7616bf1f9; no voters:
I20260810 14:44:56.104414 16688 leader_election.cc:288] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:56.104533 16693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:56.104811 16693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [term 1 LEADER]: Becoming Leader. State: Replica: 8f75bc74c8014076b53072c7616bf1f9, State: Running, Role: LEADER
I20260810 14:44:56.105194 16693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f75bc74c8014076b53072c7616bf1f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } }
I20260810 14:44:56.105805 16688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:56.106916 16700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f75bc74c8014076b53072c7616bf1f9. Latest consensus state: current_term: 1 leader_uuid: "8f75bc74c8014076b53072c7616bf1f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f75bc74c8014076b53072c7616bf1f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } } }
I20260810 14:44:56.107046 16700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:56.107360 16703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f75bc74c8014076b53072c7616bf1f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f75bc74c8014076b53072c7616bf1f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } } }
I20260810 14:44:56.107461 16703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9 [sys.catalog]: This master's current role is: LEADER
W20260810 14:44:56.110846 16726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:44:56.110934 16726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:44:56.111366 16715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:56.112341 16715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:56.113622 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:56.105473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:33335 (local address 127.0.0.1:37347)
0810 14:44:56.105579 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:56.105583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:56.105826 (+ 243us) server_negotiation.cc:407] Connection header received
0810 14:44:56.105983 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:56.105986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:56.106034 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:56.106097 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:56.106864 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:56.107608 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:56.108793 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:56.109039 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:56.109124 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:56.110445 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:56.110449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:56.110452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:44:56.110477 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:56.111867 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:56.112323 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:56.112327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:56.112329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:56.112444 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:44:56.112754 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:44:56.112757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:44:56.112759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:44:56.112940 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:56.113067 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:56.113386 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 14:44:56.113473 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:44:56.118666 16715 catalog_manager.cc:1347] Generated new cluster ID: 9428338949484748a652908388932a53
I20260810 14:44:56.118749 16715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:56.138990 16715 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:56.140349 16715 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:56.157658 16715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f75bc74c8014076b53072c7616bf1f9: Generated new TSK 0
I20260810 14:44:56.158622 16715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:56.172995 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 16579
W20260810 14:44:56.187105 16552 connection.cc:535] client connection to 127.0.0.1:37347 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37347): Connection reset by peer (error 104)
2026-08-10T14:44:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1558 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:56 dist-test-slave-rmhg krb5kdc[16739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:56 dist-test-slave-rmhg krb5kdc[16739](info): set up 2 sockets
Aug 10 14:44:56 dist-test-slave-rmhg krb5kdc[16739](info): commencing operation
krb5kdc: starting...
W20260810 14:44:57.333916 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:57 dist-test-slave-rmhg krb5kdc[16739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:44:57.468571 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44511
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:57.583076 16758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:57.583387 16758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:57.583449 16758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:57.587560 16758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:57.587641 16758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:57.587670 16758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:57.587692 16758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:57.587711 16758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:57.593264 16758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44511
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:57.594480 16758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:57.595577 16758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:57.604296 16770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:57.604158 16768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:57.604339 16758 server_base.cc:1034] running on GCE node
W20260810 14:44:57.604224 16776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:57.605127 16758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:57.606344 16758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:57.607492 16758 hybrid_clock.cc:648] HybridClock initialized: now 1786373097607469 us; error 39 us; skew 500 ppm
Aug 10 14:44:57 dist-test-slave-rmhg krb5kdc[16739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:57.614887 16758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:44:57.615897 16758 webserver.cc:460] Webserver started at http://127.0.0.1:42193/ using document root <none> and password file <none>
I20260810 14:44:57.616477 16758 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:57.616546 16758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:57.616756 16758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:57.618903 16758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "bc7d6131b5fd4a03946665390aefc06f"
format_stamp: "Formatted at 2026-08-10 14:44:57 on dist-test-slave-rmhg"
server_key: "f4a32866013c72cebbfff1c01abe1140"
server_key_iv: "474e537ac345904c74bfb5d0218a0ec6"
server_key_version: "encryptionkey@0"
I20260810 14:44:57.619454 16758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "bc7d6131b5fd4a03946665390aefc06f"
format_stamp: "Formatted at 2026-08-10 14:44:57 on dist-test-slave-rmhg"
server_key: "f4a32866013c72cebbfff1c01abe1140"
server_key_iv: "474e537ac345904c74bfb5d0218a0ec6"
server_key_version: "encryptionkey@0"
I20260810 14:44:57.623279 16758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 14:44:57.625936 16783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:57.627213 16758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:44:57.627336 16758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "bc7d6131b5fd4a03946665390aefc06f"
format_stamp: "Formatted at 2026-08-10 14:44:57 on dist-test-slave-rmhg"
server_key: "f4a32866013c72cebbfff1c01abe1140"
server_key_iv: "474e537ac345904c74bfb5d0218a0ec6"
server_key_version: "encryptionkey@0"
I20260810 14:44:57.627442 16758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:57.648082 16758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:57.651013 16758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:57.651188 16758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:57.658344 16758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44511
I20260810 14:44:57.658398 16854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44511 every 8 connection(s)
I20260810 14:44:57.659353 16758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:57.662317 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:57.666085 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 16758
I20260810 14:44:57.666258 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:57.666548 15696 external_mini_cluster.cc:1428] Setting key de89024c2b1658e491d5dbea30943b6a
I20260810 14:44:57.668824 16855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f: Bootstrap starting.
I20260810 14:44:57.671182 16855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:57.671926 16855 log.cc:824] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:57.674207 16855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f: No bootstrap required, opened a new log
I20260810 14:44:57.676184 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:57.668086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:46013 (local address 127.0.0.1:44511)
0810 14:44:57.668578 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:57.668594 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:57.668635 (+ 41us) server_negotiation.cc:407] Connection header received
0810 14:44:57.669430 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:57.669453 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:57.669823 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:57.670172 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:57.670604 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:57.671665 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:57.672618 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:57.673123 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:57.673310 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:57.674249 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:57.674255 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:57.674269 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:57.674694 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:57.674749 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:57.674772 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:57.674897 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:57.675105 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 14:44:57.675243 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":144,"threads_started":1}
I20260810 14:44:57.677453 16855 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7d6131b5fd4a03946665390aefc06f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44511 } }
I20260810 14:44:57.677739 16855 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:57.677798 16855 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7d6131b5fd4a03946665390aefc06f, State: Initialized, Role: FOLLOWER
I20260810 14:44:57.678267 16855 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7d6131b5fd4a03946665390aefc06f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44511 } }
I20260810 14:44:57.678406 16855 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:57.678464 16855 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:57.678565 16855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:57.679308 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:44:57.679628 16855 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7d6131b5fd4a03946665390aefc06f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44511 } }
I20260810 14:44:57.679962 16855 leader_election.cc:302] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc7d6131b5fd4a03946665390aefc06f; no voters:
I20260810 14:44:57.680286 16855 leader_election.cc:288] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:57.680382 16866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:57.680557 16866 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [term 1 LEADER]: Becoming Leader. State: Replica: bc7d6131b5fd4a03946665390aefc06f, State: Running, Role: LEADER
I20260810 14:44:57.680927 16866 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7d6131b5fd4a03946665390aefc06f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44511 } }
I20260810 14:44:57.681670 16855 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:57.682246 16872 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7d6131b5fd4a03946665390aefc06f. Latest consensus state: current_term: 1 leader_uuid: "bc7d6131b5fd4a03946665390aefc06f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7d6131b5fd4a03946665390aefc06f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44511 } } }
I20260810 14:44:57.682441 16872 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:57.682660 16869 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc7d6131b5fd4a03946665390aefc06f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7d6131b5fd4a03946665390aefc06f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44511 } } }
I20260810 14:44:57.682780 16869 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:57.686789 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:57.681507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:58345 (local address 127.0.0.1:44511)
0810 14:44:57.681627 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:57.681631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:57.682387 (+ 756us) server_negotiation.cc:407] Connection header received
0810 14:44:57.682589 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:57.682592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:57.682646 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:57.682701 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:57.683136 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:57.683896 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:57.684944 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:44:57.685306 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:44:57.685416 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:44:57.686317 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:57.686320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:57.686323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:57.686338 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:57.686369 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:57.686382 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:57.686469 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:57.686627 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 14:44:57.686663 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 14:44:57.686874 16897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:44:57.686941 16897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:44:57.687013 16884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:57.687845 16884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:57.693794 16884 catalog_manager.cc:1347] Generated new cluster ID: 236f8f91679542859b1133a759cf3ef1
I20260810 14:44:57.693861 16884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:57.710013 16884 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:57.711072 16884 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:57.725570 16884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc7d6131b5fd4a03946665390aefc06f: Generated new TSK 0
I20260810 14:44:57.726372 16884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:57.738791 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 16758
W20260810 14:44:57.754822 16732 connection.cc:535] client connection to 127.0.0.1:44511 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44511): Connection reset by peer (error 104)
2026-08-10T14:44:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1567 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:57 dist-test-slave-rmhg krb5kdc[16910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:57 dist-test-slave-rmhg krb5kdc[16910](info): set up 2 sockets
Aug 10 14:44:57 dist-test-slave-rmhg krb5kdc[16910](info): commencing operation
krb5kdc: starting...
W20260810 14:44:58.896819 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:44:58 dist-test-slave-rmhg krb5kdc[16910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:44:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:44:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:44:59.031934 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:44:59.150606 16934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:44:59.150902 16934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:44:59.150964 16934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:44:59.155092 16934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:44:59.155170 16934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:44:59.155200 16934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:44:59.155220 16934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:44:59.155241 16934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:44:59.160765 16934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:44:59.162003 16934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:44:59.163122 16934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:44:59.172636 16942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:59.172811 16934 server_base.cc:1034] running on GCE node
W20260810 14:44:59.172689 16945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:44:59.172731 16943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:44:59.173581 16934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:44:59.174829 16934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:44:59.176005 16934 hybrid_clock.cc:648] HybridClock initialized: now 1786373099175992 us; error 31 us; skew 500 ppm
Aug 10 14:44:59 dist-test-slave-rmhg krb5kdc[16910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:44:59.178930 16934 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:44:59.179916 16934 webserver.cc:460] Webserver started at http://127.0.0.1:41129/ using document root <none> and password file <none>
I20260810 14:44:59.180485 16934 fs_manager.cc:359] Metadata directory not provided
I20260810 14:44:59.180554 16934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:44:59.180768 16934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:44:59.182992 16934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "04412251fe174bdba2435ca3a7a5a228"
format_stamp: "Formatted at 2026-08-10 14:44:59 on dist-test-slave-rmhg"
server_key: "802cf312f351063cebb07b5d9774a245"
server_key_iv: "b5711e36c7ed15f241b3ebacefd2c9db"
server_key_version: "encryptionkey@0"
I20260810 14:44:59.183578 16934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "04412251fe174bdba2435ca3a7a5a228"
format_stamp: "Formatted at 2026-08-10 14:44:59 on dist-test-slave-rmhg"
server_key: "802cf312f351063cebb07b5d9774a245"
server_key_iv: "b5711e36c7ed15f241b3ebacefd2c9db"
server_key_version: "encryptionkey@0"
I20260810 14:44:59.187507 16934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 14:44:59.190060 16954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:44:59.191285 16934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:44:59.191416 16934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "04412251fe174bdba2435ca3a7a5a228"
format_stamp: "Formatted at 2026-08-10 14:44:59 on dist-test-slave-rmhg"
server_key: "802cf312f351063cebb07b5d9774a245"
server_key_iv: "b5711e36c7ed15f241b3ebacefd2c9db"
server_key_version: "encryptionkey@0"
I20260810 14:44:59.191517 16934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:44:59.196238 16934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:44:59.199532 16934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:44:59.199723 16934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:44:59.207396 16934 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39641
I20260810 14:44:59.207410 17025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39641 every 8 connection(s)
I20260810 14:44:59.208408 16934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:44:59.209597 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 16934
I20260810 14:44:59.209740 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:44:59.210026 15696 external_mini_cluster.cc:1428] Setting key aa06d938d97b2c16c19a5177bd5e886f
I20260810 14:44:59.211763 17027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:44:59.215739 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:59.211535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:40171 (local address 127.0.0.1:39641)
0810 14:44:59.212109 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:59.212130 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:59.212172 (+ 42us) server_negotiation.cc:407] Connection header received
0810 14:44:59.212863 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:59.212881 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:59.213251 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:59.213659 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:59.214453 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:59.214458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:59.214482 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:59.214493 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:59.214552 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:59.214573 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:59.214708 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:59.214811 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 14:44:59.214995 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":173,"threads_started":1}
I20260810 14:44:59.217893 17027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228: Bootstrap starting.
I20260810 14:44:59.218688 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:44:59.220324 17027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228: Neither blocks nor log segments found. Creating new log.
I20260810 14:44:59.221277 17027 log.cc:824] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228: Log is configured to *not* fsync() on all Append() calls
I20260810 14:44:59.222538 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:44:59.220783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:40919 (local address 127.0.0.1:39641)
0810 14:44:59.220931 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:44:59.220935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:44:59.221135 (+ 200us) server_negotiation.cc:407] Connection header received
0810 14:44:59.221283 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:44:59.221287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:44:59.221322 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:44:59.221406 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:44:59.222202 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:44:59.222206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:44:59.222209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:44:59.222220 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:44:59.222248 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:44:59.222261 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:44:59.222352 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 14:44:59.222390 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 14:44:59.222427 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:44:59.223524 17027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228: No bootstrap required, opened a new log
I20260810 14:44:59.227095 17027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04412251fe174bdba2435ca3a7a5a228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } }
I20260810 14:44:59.227334 17027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:44:59.227399 17027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04412251fe174bdba2435ca3a7a5a228, State: Initialized, Role: FOLLOWER
I20260810 14:44:59.227815 17027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04412251fe174bdba2435ca3a7a5a228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } }
I20260810 14:44:59.227945 17027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:44:59.228012 17027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:44:59.228116 17027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:44:59.228996 17027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04412251fe174bdba2435ca3a7a5a228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } }
I20260810 14:44:59.229359 17027 leader_election.cc:302] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04412251fe174bdba2435ca3a7a5a228; no voters:
I20260810 14:44:59.229652 17027 leader_election.cc:288] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:44:59.229785 17044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:44:59.230021 17044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [term 1 LEADER]: Becoming Leader. State: Replica: 04412251fe174bdba2435ca3a7a5a228, State: Running, Role: LEADER
I20260810 14:44:59.230355 17044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04412251fe174bdba2435ca3a7a5a228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } }
I20260810 14:44:59.231078 17027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:44:59.231923 17047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04412251fe174bdba2435ca3a7a5a228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04412251fe174bdba2435ca3a7a5a228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } } }
I20260810 14:44:59.232089 17047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:59.232549 17062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:44:59.232595 17049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04412251fe174bdba2435ca3a7a5a228. Latest consensus state: current_term: 1 leader_uuid: "04412251fe174bdba2435ca3a7a5a228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04412251fe174bdba2435ca3a7a5a228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } } }
I20260810 14:44:59.232738 17049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228 [sys.catalog]: This master's current role is: LEADER
I20260810 14:44:59.235849 17062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:44:59.242380 17062 catalog_manager.cc:1347] Generated new cluster ID: 992099fb9f0940998033876f0a6b1e42
I20260810 14:44:59.242460 17062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:44:59.264928 17062 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:44:59.265358 17062 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:44:59.271186 17062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04412251fe174bdba2435ca3a7a5a228: Generated new TSK 0
I20260810 14:44:59.272095 17062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:44:59.275496 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 16934
2026-08-10T14:44:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1532 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:44:59 dist-test-slave-rmhg krb5kdc[17083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:44:59 dist-test-slave-rmhg krb5kdc[17083](info): set up 2 sockets
Aug 10 14:44:59 dist-test-slave-rmhg krb5kdc[17083](info): commencing operation
krb5kdc: starting...
W20260810 14:45:00.440980 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:45:00 dist-test-slave-rmhg krb5kdc[17083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:45:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:45:00.576763 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:44089
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:44089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:45:00.692068 17105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:45:00.692369 17105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:45:00.692428 17105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:45:00.696519 17105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:45:00.696596 17105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:45:00.696622 17105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:45:00.696642 17105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:45:00.696661 17105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:45:00.702189 17105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:44089
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:44089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:45:00.703389 17105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:45:00.704507 17105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:45:00.713476 17113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:00.713497 17115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:00.713469 17117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:00.714017 17105 server_base.cc:1034] running on GCE node
I20260810 14:45:00.714640 17105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:45:00.715828 17105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:45:00.717026 17105 hybrid_clock.cc:648] HybridClock initialized: now 1786373100717005 us; error 35 us; skew 500 ppm
Aug 10 14:45:00 dist-test-slave-rmhg krb5kdc[17083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373100, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:45:00.719914 17105 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:45:00.721040 17105 webserver.cc:460] Webserver started at http://127.15.84.62:34023/ using document root <none> and password file <none>
I20260810 14:45:00.721791 17105 fs_manager.cc:359] Metadata directory not provided
I20260810 14:45:00.721894 17105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:45:00.722175 17105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:45:00.725329 17105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "2e8c562a2f1d48e9b6f3d950cd962531"
format_stamp: "Formatted at 2026-08-10 14:45:00 on dist-test-slave-rmhg"
server_key: "58036e579c437cf291f63c55ec285836"
server_key_iv: "9287b2d3ec1ba45e0b094a90a092b4a3"
server_key_version: "encryptionkey@0"
I20260810 14:45:00.726091 17105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "2e8c562a2f1d48e9b6f3d950cd962531"
format_stamp: "Formatted at 2026-08-10 14:45:00 on dist-test-slave-rmhg"
server_key: "58036e579c437cf291f63c55ec285836"
server_key_iv: "9287b2d3ec1ba45e0b094a90a092b4a3"
server_key_version: "encryptionkey@0"
I20260810 14:45:00.731810 17105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 14:45:00.734633 17125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:45:00.735987 17105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:45:00.736119 17105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "2e8c562a2f1d48e9b6f3d950cd962531"
format_stamp: "Formatted at 2026-08-10 14:45:00 on dist-test-slave-rmhg"
server_key: "58036e579c437cf291f63c55ec285836"
server_key_iv: "9287b2d3ec1ba45e0b094a90a092b4a3"
server_key_version: "encryptionkey@0"
I20260810 14:45:00.736236 17105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:45:00.759069 17105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:45:00.762269 17105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:45:00.762503 17105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:45:00.770637 17105 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:44089
I20260810 14:45:00.770664 17214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:44089 every 8 connection(s)
I20260810 14:45:00.771687 17105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:45:00.774700 17215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:45:00.775121 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 17105
I20260810 14:45:00.775336 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:45:00.775662 15696 external_mini_cluster.cc:1428] Setting key 7229447db66956d8bbdc167fc602721c
I20260810 14:45:00.782300 17215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531: Bootstrap starting.
I20260810 14:45:00.785319 17215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531: Neither blocks nor log segments found. Creating new log.
I20260810 14:45:00.786232 17215 log.cc:824] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531: Log is configured to *not* fsync() on all Append() calls
Aug 10 14:45:00 dist-test-slave-rmhg krb5kdc[17083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:45:00.789033 17215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531: No bootstrap required, opened a new log
I20260810 14:45:00.793138 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:00.777302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46373 (local address 127.15.84.62:44089)
0810 14:45:00.777815 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:00.777836 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:00.777887 (+ 51us) server_negotiation.cc:407] Connection header received
0810 14:45:00.778745 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:00.778774 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:00.779263 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:00.779688 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:00.780518 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:00.782109 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:00.783145 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:00.783902 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:00.784081 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:00.787168 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:00.787176 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:00.787198 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:00.787232 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:00.789320 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:00.789930 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:00.789937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:00.789950 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:00.790107 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:00.790556 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:00.790560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:00.790562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:00.791149 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:00.791382 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:00.791769 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 14:45:00.792037 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":146,"threads_started":1}
I20260810 14:45:00.793918 17215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e8c562a2f1d48e9b6f3d950cd962531" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44089 } }
I20260810 14:45:00.794236 17215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:45:00.794297 17215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e8c562a2f1d48e9b6f3d950cd962531, State: Initialized, Role: FOLLOWER
I20260810 14:45:00.794746 17215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e8c562a2f1d48e9b6f3d950cd962531" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44089 } }
I20260810 14:45:00.794873 17215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:45:00.794929 17215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:45:00.795032 17215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:45:00.796183 17215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e8c562a2f1d48e9b6f3d950cd962531" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44089 } }
I20260810 14:45:00.796561 17215 leader_election.cc:302] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e8c562a2f1d48e9b6f3d950cd962531; no voters:
I20260810 14:45:00.796693 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:45:00.796907 17215 leader_election.cc:288] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:45:00.797029 17224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:45:00.797325 17224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [term 1 LEADER]: Becoming Leader. State: Replica: 2e8c562a2f1d48e9b6f3d950cd962531, State: Running, Role: LEADER
I20260810 14:45:00.797683 17224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e8c562a2f1d48e9b6f3d950cd962531" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44089 } }
I20260810 14:45:00.798259 17215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:45:00.799010 17224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e8c562a2f1d48e9b6f3d950cd962531. Latest consensus state: current_term: 1 leader_uuid: "2e8c562a2f1d48e9b6f3d950cd962531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e8c562a2f1d48e9b6f3d950cd962531" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44089 } } }
I20260810 14:45:00.799147 17224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [sys.catalog]: This master's current role is: LEADER
I20260810 14:45:00.799789 17225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e8c562a2f1d48e9b6f3d950cd962531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e8c562a2f1d48e9b6f3d950cd962531" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 44089 } } }
I20260810 14:45:00.799890 17225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531 [sys.catalog]: This master's current role is: LEADER
W20260810 14:45:00.803890 17262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:45:00.804001 17262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:45:00.804075 17254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:45:00.804924 17254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:45:00.809613 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:00.800056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33169 (local address 127.15.84.62:44089)
0810 14:45:00.800176 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:00.800180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:00.800462 (+ 282us) server_negotiation.cc:407] Connection header received
0810 14:45:00.800654 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:00.800657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:00.800717 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:00.800764 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:00.801715 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:00.802478 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:00.803654 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:00.803985 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:00.804094 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:00.806065 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:00.806082 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:00.806084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:00.806115 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:00.807703 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:00.808238 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:00.808242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:00.808243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:00.808326 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:00.808664 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:00.808668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:00.808670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:00.808861 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:00.808995 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:00.809342 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 14:45:00.809446 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:45:00.811633 17254 catalog_manager.cc:1347] Generated new cluster ID: cae5527fb60d451e8791ed6bc302d1f4
I20260810 14:45:00.811717 17254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:45:00.828544 17254 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:45:00.829644 17254 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:45:00.837515 17254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e8c562a2f1d48e9b6f3d950cd962531: Generated new TSK 0
I20260810 14:45:00.838490 17254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:45:00.845808 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 17105
W20260810 14:45:00.863695 17081 connection.cc:535] client connection to 127.15.84.62:44089 recv error: Network error: failed to read from TLS socket (remote: 127.15.84.62:44089): Connection reset by peer (error 104)
2026-08-10T14:45:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1575 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:45:00 dist-test-slave-rmhg krb5kdc[17275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:45:00 dist-test-slave-rmhg krb5kdc[17275](info): set up 2 sockets
Aug 10 14:45:00 dist-test-slave-rmhg krb5kdc[17275](info): commencing operation
krb5kdc: starting...
W20260810 14:45:02.022269 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:45:02 dist-test-slave-rmhg krb5kdc[17275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:45:02.160550 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:45515
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:45515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:45:02.277503 17295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:45:02.277807 17295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:45:02.277869 17295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:45:02.282191 17295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:45:02.282275 17295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:45:02.282303 17295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:45:02.282326 17295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:45:02.282353 17295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:45:02.287982 17295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:45515
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:45515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:45:02.289132 17295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:45:02.290244 17295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:45:02.299557 17295 server_base.cc:1034] running on GCE node
W20260810 14:45:02.299496 17306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:02.299672 17305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:02.299872 17310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:02.300477 17295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:45:02.301772 17295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:45:02.302994 17295 hybrid_clock.cc:648] HybridClock initialized: now 1786373102302961 us; error 54 us; skew 500 ppm
Aug 10 14:45:02 dist-test-slave-rmhg krb5kdc[17275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373102, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:45:02.306026 17295 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:45:02.307051 17295 webserver.cc:460] Webserver started at http://127.15.84.62:36345/ using document root <none> and password file <none>
I20260810 14:45:02.307646 17295 fs_manager.cc:359] Metadata directory not provided
I20260810 14:45:02.307742 17295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:45:02.307968 17295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:45:02.310304 17295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "5c1d59b5279b491394fe264448cb498a"
format_stamp: "Formatted at 2026-08-10 14:45:02 on dist-test-slave-rmhg"
server_key: "7d04a10d01b7b29d5bb98fbf8b71db29"
server_key_iv: "366de6a045d99b2218cddf5776cc02da"
server_key_version: "encryptionkey@0"
I20260810 14:45:02.310921 17295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "5c1d59b5279b491394fe264448cb498a"
format_stamp: "Formatted at 2026-08-10 14:45:02 on dist-test-slave-rmhg"
server_key: "7d04a10d01b7b29d5bb98fbf8b71db29"
server_key_iv: "366de6a045d99b2218cddf5776cc02da"
server_key_version: "encryptionkey@0"
I20260810 14:45:02.315214 17295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:45:02.318166 17322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:45:02.319649 17295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:45:02.319834 17295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "5c1d59b5279b491394fe264448cb498a"
format_stamp: "Formatted at 2026-08-10 14:45:02 on dist-test-slave-rmhg"
server_key: "7d04a10d01b7b29d5bb98fbf8b71db29"
server_key_iv: "366de6a045d99b2218cddf5776cc02da"
server_key_version: "encryptionkey@0"
I20260810 14:45:02.319963 17295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:45:02.342828 17295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:45:02.346359 17295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:45:02.346558 17295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:45:02.353822 17295 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:45515
I20260810 14:45:02.353845 17405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:45515 every 8 connection(s)
I20260810 14:45:02.354924 17295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:45:02.357903 17406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:45:02.358812 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 17295
I20260810 14:45:02.358955 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:45:02.359227 15696 external_mini_cluster.cc:1428] Setting key 572e8b272b9d98b77193a595a15bf103
I20260810 14:45:02.364241 17406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a: Bootstrap starting.
I20260810 14:45:02.366667 17406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a: Neither blocks nor log segments found. Creating new log.
I20260810 14:45:02.367427 17406 log.cc:824] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:45:02.368556 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:02.360657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57349 (local address 127.15.84.62:45515)
0810 14:45:02.361125 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:02.361141 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:02.361212 (+ 71us) server_negotiation.cc:407] Connection header received
0810 14:45:02.361874 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:02.361893 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:02.362270 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:02.362595 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:02.363036 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:02.364061 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:02.365045 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:02.365602 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:02.365754 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:02.366663 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:02.366670 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:02.366685 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:45:02.367118 (+ 433us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:02.367185 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:45:02.367211 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:02.367327 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:02.367690 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 14:45:02.367815 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":141,"threads_started":1}
I20260810 14:45:02.369517 17406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a: No bootstrap required, opened a new log
I20260810 14:45:02.371950 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:45:02.373807 17406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1d59b5279b491394fe264448cb498a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45515 } }
I20260810 14:45:02.374117 17406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:45:02.374202 17406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1d59b5279b491394fe264448cb498a, State: Initialized, Role: FOLLOWER
I20260810 14:45:02.374756 17406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1d59b5279b491394fe264448cb498a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45515 } }
I20260810 14:45:02.374944 17406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:45:02.375025 17406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:45:02.375149 17406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:45:02.376348 17406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1d59b5279b491394fe264448cb498a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45515 } }
I20260810 14:45:02.376731 17406 leader_election.cc:302] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c1d59b5279b491394fe264448cb498a; no voters:
I20260810 14:45:02.377107 17406 leader_election.cc:288] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:45:02.377555 17426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:45:02.377821 17426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [term 1 LEADER]: Becoming Leader. State: Replica: 5c1d59b5279b491394fe264448cb498a, State: Running, Role: LEADER
I20260810 14:45:02.378553 17426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1d59b5279b491394fe264448cb498a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45515 } }
I20260810 14:45:02.378683 17406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:45:02.380016 17426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c1d59b5279b491394fe264448cb498a. Latest consensus state: current_term: 1 leader_uuid: "5c1d59b5279b491394fe264448cb498a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1d59b5279b491394fe264448cb498a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45515 } } }
I20260810 14:45:02.380219 17426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [sys.catalog]: This master's current role is: LEADER
I20260810 14:45:02.380419 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:02.374122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46183 (local address 127.15.84.62:45515)
0810 14:45:02.374247 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:02.374251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:02.374517 (+ 266us) server_negotiation.cc:407] Connection header received
0810 14:45:02.374637 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:02.374639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:02.374686 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:02.374754 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:02.375126 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:02.375856 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:02.377071 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:02.377460 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:02.377991 (+ 531us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:02.378634 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:02.378638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:02.378641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:45:02.378655 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:02.378686 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:45:02.378699 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:02.380177 (+ 1478us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:02.380233 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 14:45:02.380274 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:45:02.382033 17423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c1d59b5279b491394fe264448cb498a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1d59b5279b491394fe264448cb498a" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 45515 } } }
I20260810 14:45:02.382138 17446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:45:02.382279 17423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a [sys.catalog]: This master's current role is: LEADER
I20260810 14:45:02.385042 17446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:45:02.390450 17446 catalog_manager.cc:1347] Generated new cluster ID: 908266ecf3d24b769074f71e482fe12e
I20260810 14:45:02.390530 17446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:45:02.409749 17446 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:45:02.411173 17446 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:45:02.425796 17446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c1d59b5279b491394fe264448cb498a: Generated new TSK 0
I20260810 14:45:02.426798 17446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:45:02.439831 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 17295
W20260810 14:45:02.452556 17270 connection.cc:535] client connection to 127.15.84.62:45515 recv error: Network error: failed to read from TLS socket (remote: 127.15.84.62:45515): Connection reset by peer (error 104)
2026-08-10T14:45:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1589 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:45:02 dist-test-slave-rmhg krb5kdc[17458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:45:02 dist-test-slave-rmhg krb5kdc[17458](info): set up 2 sockets
Aug 10 14:45:02 dist-test-slave-rmhg krb5kdc[17458](info): commencing operation
krb5kdc: starting...
W20260810 14:45:03.607460 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:45:03 dist-test-slave-rmhg krb5kdc[17458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:45:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:45:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:45:03.751610 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:32773
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:35607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:32773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:45:03.867857 17481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:45:03.868155 17481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:45:03.868223 17481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:45:03.872299 17481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:45:03.872386 17481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:45:03.872411 17481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:45:03.872432 17481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:45:03.872450 17481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:45:03.877926 17481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:35607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:32773
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:32773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:45:03.879050 17481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:45:03.880096 17481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:45:03.888757 17493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:03.888761 17497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:03.888761 17495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:03.889104 17481 server_base.cc:1034] running on GCE node
I20260810 14:45:03.889735 17481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:45:03.890885 17481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:45:03.892047 17481 hybrid_clock.cc:648] HybridClock initialized: now 1786373103892022 us; error 40 us; skew 500 ppm
Aug 10 14:45:03 dist-test-slave-rmhg krb5kdc[17458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373103, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:45:03.894941 17481 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:45:03.895951 17481 webserver.cc:460] Webserver started at http://127.15.84.62:44211/ using document root <none> and password file <none>
I20260810 14:45:03.896533 17481 fs_manager.cc:359] Metadata directory not provided
I20260810 14:45:03.896595 17481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:45:03.896801 17481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:45:03.898967 17481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "855f782bf20a4dac8b1a771dee9fdaaf"
format_stamp: "Formatted at 2026-08-10 14:45:03 on dist-test-slave-rmhg"
server_key: "c179a1668b2bbf183a1e609aa2059170"
server_key_iv: "717012a497d824df0818615a96aeff9c"
server_key_version: "encryptionkey@0"
I20260810 14:45:03.899515 17481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "855f782bf20a4dac8b1a771dee9fdaaf"
format_stamp: "Formatted at 2026-08-10 14:45:03 on dist-test-slave-rmhg"
server_key: "c179a1668b2bbf183a1e609aa2059170"
server_key_iv: "717012a497d824df0818615a96aeff9c"
server_key_version: "encryptionkey@0"
I20260810 14:45:03.903446 17481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 14:45:03.906018 17508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:45:03.907299 17481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:45:03.907425 17481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "855f782bf20a4dac8b1a771dee9fdaaf"
format_stamp: "Formatted at 2026-08-10 14:45:03 on dist-test-slave-rmhg"
server_key: "c179a1668b2bbf183a1e609aa2059170"
server_key_iv: "717012a497d824df0818615a96aeff9c"
server_key_version: "encryptionkey@0"
I20260810 14:45:03.907531 17481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:45:03.912441 17481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:45:03.915715 17481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:45:03.915916 17481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:45:03.923195 17481 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:32773
I20260810 14:45:03.923209 17578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:32773 every 8 connection(s)
I20260810 14:45:03.924207 17481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:45:03.927287 17579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:45:03.929272 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 17481
I20260810 14:45:03.929453 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:45:03.929776 15696 external_mini_cluster.cc:1428] Setting key eb538b4ca101953210344ab0882fbb5a
I20260810 14:45:03.934692 17579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf: Bootstrap starting.
I20260810 14:45:03.935696 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:03.931519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53733 (local address 127.15.84.62:32773)
0810 14:45:03.932047 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:03.932064 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:03.932105 (+ 41us) server_negotiation.cc:407] Connection header received
0810 14:45:03.932809 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:03.932828 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:03.933139 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:03.933533 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:03.934392 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:03.934397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:03.934410 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:45:03.934421 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:03.934479 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:45:03.934499 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:03.934631 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:03.934722 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 14:45:03.934893 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":160,"threads_started":1}
I20260810 14:45:03.937448 17579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf: Neither blocks nor log segments found. Creating new log.
I20260810 14:45:03.938387 17579 log.cc:824] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf: Log is configured to *not* fsync() on all Append() calls
I20260810 14:45:03.938874 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:45:03.940632 17579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf: No bootstrap required, opened a new log
I20260810 14:45:03.942649 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:03.940899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41759 (local address 127.15.84.62:32773)
0810 14:45:03.941012 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:03.941016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:03.941261 (+ 245us) server_negotiation.cc:407] Connection header received
0810 14:45:03.941402 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:03.941406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:03.941443 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:03.941511 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:03.942273 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:03.942277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:03.942280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:45:03.942292 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:03.942321 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:45:03.942332 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:03.942406 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:03.942486 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 14:45:03.942527 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 14:45:03.944941 17579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855f782bf20a4dac8b1a771dee9fdaaf" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 32773 } }
I20260810 14:45:03.945304 17579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:45:03.945374 17579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 855f782bf20a4dac8b1a771dee9fdaaf, State: Initialized, Role: FOLLOWER
I20260810 14:45:03.945851 17579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855f782bf20a4dac8b1a771dee9fdaaf" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 32773 } }
I20260810 14:45:03.946014 17579 raft_consensus.cc:388] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:45:03.946086 17579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:45:03.946195 17579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:45:03.947192 17579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855f782bf20a4dac8b1a771dee9fdaaf" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 32773 } }
I20260810 14:45:03.947546 17579 leader_election.cc:302] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 855f782bf20a4dac8b1a771dee9fdaaf; no voters:
I20260810 14:45:03.947841 17579 leader_election.cc:288] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:45:03.947966 17595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:45:03.948195 17595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [term 1 LEADER]: Becoming Leader. State: Replica: 855f782bf20a4dac8b1a771dee9fdaaf, State: Running, Role: LEADER
I20260810 14:45:03.948542 17595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855f782bf20a4dac8b1a771dee9fdaaf" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 32773 } }
I20260810 14:45:03.949219 17579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:45:03.950227 17597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 855f782bf20a4dac8b1a771dee9fdaaf. Latest consensus state: current_term: 1 leader_uuid: "855f782bf20a4dac8b1a771dee9fdaaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855f782bf20a4dac8b1a771dee9fdaaf" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 32773 } } }
I20260810 14:45:03.950356 17597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [sys.catalog]: This master's current role is: LEADER
I20260810 14:45:03.950836 17596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "855f782bf20a4dac8b1a771dee9fdaaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855f782bf20a4dac8b1a771dee9fdaaf" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 32773 } } }
I20260810 14:45:03.950994 17596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf [sys.catalog]: This master's current role is: LEADER
W20260810 14:45:03.954149 17616 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:45:03.954228 17616 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:45:03.954320 17608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:45:03.955045 17608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:45:03.961455 17608 catalog_manager.cc:1347] Generated new cluster ID: f9931c79d93c41deaa5716537b3eb408
I20260810 14:45:03.961545 17608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:45:03.988744 17608 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:45:03.989183 17608 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:45:03.995440 17608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 855f782bf20a4dac8b1a771dee9fdaaf: Generated new TSK 0
I20260810 14:45:03.996304 17608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:45:04.005290 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 17481
2026-08-10T14:45:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1569 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:45:04 dist-test-slave-rmhg krb5kdc[17629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:45:04 dist-test-slave-rmhg krb5kdc[17629](info): set up 2 sockets
Aug 10 14:45:04 dist-test-slave-rmhg krb5kdc[17629](info): commencing operation
krb5kdc: starting...
W20260810 14:45:05.163334 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:45:05 dist-test-slave-rmhg krb5kdc[17629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:45:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:45:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:45:05.295558 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40443
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:39853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:40443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:45:05.410899 17648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:45:05.411226 17648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:45:05.411288 17648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:45:05.415577 17648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:45:05.415659 17648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:45:05.415695 17648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:45:05.415727 17648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:45:05.415755 17648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:45:05.421329 17648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:39853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:40443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:40443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:45:05.422502 17648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:45:05.423593 17648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:45:05.432662 17657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:05.432608 17659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:05.432682 17648 server_base.cc:1034] running on GCE node
W20260810 14:45:05.432610 17656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:05.433522 17648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:45:05.434801 17648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:45:05.435984 17648 hybrid_clock.cc:648] HybridClock initialized: now 1786373105435955 us; error 47 us; skew 500 ppm
Aug 10 14:45:05 dist-test-slave-rmhg krb5kdc[17629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373105, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:45:05.439090 17648 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:45:05.440162 17648 webserver.cc:460] Webserver started at http://127.15.84.62:33081/ using document root <none> and password file <none>
I20260810 14:45:05.440775 17648 fs_manager.cc:359] Metadata directory not provided
I20260810 14:45:05.440850 17648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:45:05.441066 17648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:45:05.443858 17648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "971ce09560054e119f7686d187b0ed6b"
format_stamp: "Formatted at 2026-08-10 14:45:05 on dist-test-slave-rmhg"
server_key: "2d83a3185e7b90ab6a6c256a0ff3e51d"
server_key_iv: "5a972c55787d34c411b8b762589348e1"
server_key_version: "encryptionkey@0"
I20260810 14:45:05.444509 17648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "971ce09560054e119f7686d187b0ed6b"
format_stamp: "Formatted at 2026-08-10 14:45:05 on dist-test-slave-rmhg"
server_key: "2d83a3185e7b90ab6a6c256a0ff3e51d"
server_key_iv: "5a972c55787d34c411b8b762589348e1"
server_key_version: "encryptionkey@0"
I20260810 14:45:05.448480 17648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:45:05.451188 17670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:45:05.452538 17648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 14:45:05.452693 17648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "971ce09560054e119f7686d187b0ed6b"
format_stamp: "Formatted at 2026-08-10 14:45:05 on dist-test-slave-rmhg"
server_key: "2d83a3185e7b90ab6a6c256a0ff3e51d"
server_key_iv: "5a972c55787d34c411b8b762589348e1"
server_key_version: "encryptionkey@0"
I20260810 14:45:05.452807 17648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:45:05.486824 17648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:45:05.490090 17648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:45:05.490321 17648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:45:05.497828 17648 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:40443
I20260810 14:45:05.497843 17747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:40443 every 8 connection(s)
I20260810 14:45:05.498893 17648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:45:05.501952 17748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:45:05.503876 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 17648
I20260810 14:45:05.504042 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:45:05.504359 15696 external_mini_cluster.cc:1428] Setting key 07a989327451ba8140460f4025d9cf37
I20260810 14:45:05.509658 17748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b: Bootstrap starting.
I20260810 14:45:05.512537 17748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b: Neither blocks nor log segments found. Creating new log.
I20260810 14:45:05.513561 17748 log.cc:824] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b: Log is configured to *not* fsync() on all Append() calls
Aug 10 14:45:05 dist-test-slave-rmhg krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:45:05.516252 17748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b: No bootstrap required, opened a new log
I20260810 14:45:05.519836 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:05.505816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56017 (local address 127.15.84.62:40443)
0810 14:45:05.506260 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:05.506276 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:05.506317 (+ 41us) server_negotiation.cc:407] Connection header received
0810 14:45:05.507008 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:05.507027 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:05.507411 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:05.507766 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:05.508614 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:05.509709 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:05.510921 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:05.511408 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:05.511533 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:05.514486 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:05.514493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:05.514513 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:05.514546 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:05.516455 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:05.516934 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:05.516939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:05.516946 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:05.517110 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:05.517423 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:05.517426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:05.517428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:05.517975 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:05.518200 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:05.518388 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 14:45:05.518773 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":124,"threads_started":1}
I20260810 14:45:05.521011 17748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971ce09560054e119f7686d187b0ed6b" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40443 } }
I20260810 14:45:05.521345 17748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:45:05.521409 17748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971ce09560054e119f7686d187b0ed6b, State: Initialized, Role: FOLLOWER
I20260810 14:45:05.521894 17748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971ce09560054e119f7686d187b0ed6b" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40443 } }
I20260810 14:45:05.522047 17748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:45:05.522113 17748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:45:05.522217 17748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:45:05.523090 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:45:05.523344 17748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971ce09560054e119f7686d187b0ed6b" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40443 } }
I20260810 14:45:05.523707 17748 leader_election.cc:302] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 971ce09560054e119f7686d187b0ed6b; no voters:
I20260810 14:45:05.524075 17748 leader_election.cc:288] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:45:05.524215 17759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:45:05.524451 17759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [term 1 LEADER]: Becoming Leader. State: Replica: 971ce09560054e119f7686d187b0ed6b, State: Running, Role: LEADER
I20260810 14:45:05.524844 17759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971ce09560054e119f7686d187b0ed6b" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40443 } }
I20260810 14:45:05.525548 17748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:45:05.526376 17759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 971ce09560054e119f7686d187b0ed6b. Latest consensus state: current_term: 1 leader_uuid: "971ce09560054e119f7686d187b0ed6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971ce09560054e119f7686d187b0ed6b" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40443 } } }
I20260810 14:45:05.526495 17759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [sys.catalog]: This master's current role is: LEADER
I20260810 14:45:05.527148 17766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "971ce09560054e119f7686d187b0ed6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971ce09560054e119f7686d187b0ed6b" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 40443 } } }
I20260810 14:45:05.527293 17766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b [sys.catalog]: This master's current role is: LEADER
W20260810 14:45:05.530833 17789 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:45:05.530932 17789 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:45:05.530995 17779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:45:05.531908 17779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:45:05.534806 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:05.525136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59167 (local address 127.15.84.62:40443)
0810 14:45:05.525309 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:05.525314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:05.526784 (+ 1470us) server_negotiation.cc:407] Connection header received
0810 14:45:05.526984 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:05.526988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:05.527037 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:05.527089 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:05.528163 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:05.528812 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:05.529707 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:05.529949 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:05.530044 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:05.531521 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:05.531525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:05.531528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:05.531553 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:05.532938 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:05.533433 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:05.533437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:05.533438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:05.533516 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:05.533789 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:05.533791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:05.533793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:05.533999 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:05.534131 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:05.534475 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 14:45:05.534626 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 14:45:05.538902 17779 catalog_manager.cc:1347] Generated new cluster ID: 9ed0e4ebfaa24a4c8560bef98ca465d8
I20260810 14:45:05.538985 17779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:45:05.573349 17779 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:45:05.574465 17779 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:45:05.583551 17779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 971ce09560054e119f7686d187b0ed6b: Generated new TSK 0
I20260810 14:45:05.584545 17779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:45:05.589864 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 17648
2026-08-10T14:45:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1582 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:45:05 dist-test-slave-rmhg krb5kdc[17801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:45:05 dist-test-slave-rmhg krb5kdc[17801](info): set up 2 sockets
Aug 10 14:45:05 dist-test-slave-rmhg krb5kdc[17801](info): commencing operation
krb5kdc: starting...
W20260810 14:45:06.750192 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:45:06 dist-test-slave-rmhg krb5kdc[17801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:45:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:45:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:45:06.885711 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:42215
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:42215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:45:07.005467 17821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:45:07.005793 17821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:45:07.005857 17821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:45:07.009984 17821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:45:07.010064 17821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:45:07.010090 17821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:45:07.010113 17821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:45:07.010130 17821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:45:07.015718 17821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:42215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:42215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:45:07.016906 17821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:45:07.017987 17821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:45:07.026813 17835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:07.026940 17821 server_base.cc:1034] running on GCE node
W20260810 14:45:07.026829 17828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:07.026818 17829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:07.027659 17821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:45:07.028776 17821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:45:07.029940 17821 hybrid_clock.cc:648] HybridClock initialized: now 1786373107029924 us; error 29 us; skew 500 ppm
Aug 10 14:45:07 dist-test-slave-rmhg krb5kdc[17801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373107, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:45:07.032923 17821 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:45:07.034060 17821 webserver.cc:460] Webserver started at http://127.15.84.62:44077/ using document root <none> and password file <none>
I20260810 14:45:07.034647 17821 fs_manager.cc:359] Metadata directory not provided
I20260810 14:45:07.034719 17821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:45:07.034936 17821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:45:07.037206 17821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "3c6e13a0cc694ff5b2231ae672dd55e9"
format_stamp: "Formatted at 2026-08-10 14:45:07 on dist-test-slave-rmhg"
server_key: "854dc4eee17c26112942fa71ae105770"
server_key_iv: "a4dbcead91af0ea35ec2e43f43130578"
server_key_version: "encryptionkey@0"
I20260810 14:45:07.037825 17821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "3c6e13a0cc694ff5b2231ae672dd55e9"
format_stamp: "Formatted at 2026-08-10 14:45:07 on dist-test-slave-rmhg"
server_key: "854dc4eee17c26112942fa71ae105770"
server_key_iv: "a4dbcead91af0ea35ec2e43f43130578"
server_key_version: "encryptionkey@0"
I20260810 14:45:07.041847 17821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:45:07.044601 17848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:45:07.046085 17821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:45:07.046242 17821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "3c6e13a0cc694ff5b2231ae672dd55e9"
format_stamp: "Formatted at 2026-08-10 14:45:07 on dist-test-slave-rmhg"
server_key: "854dc4eee17c26112942fa71ae105770"
server_key_iv: "a4dbcead91af0ea35ec2e43f43130578"
server_key_version: "encryptionkey@0"
I20260810 14:45:07.046372 17821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:45:07.083220 17821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:45:07.086701 17821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:45:07.086912 17821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:45:07.094789 17821 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:42215
I20260810 14:45:07.094811 17932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:42215 every 8 connection(s)
I20260810 14:45:07.095852 17821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:45:07.098933 17933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:45:07.104447 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 17821
I20260810 14:45:07.104624 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:45:07.104902 15696 external_mini_cluster.cc:1428] Setting key af67eec4cb560c3b0368d05b843a7d5a
I20260810 14:45:07.105109 17933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9: Bootstrap starting.
I20260810 14:45:07.107839 17933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9: Neither blocks nor log segments found. Creating new log.
I20260810 14:45:07.108776 17933 log.cc:824] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:45:07.111218 17933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9: No bootstrap required, opened a new log
I20260810 14:45:07.114656 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:07.106319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41285 (local address 127.15.84.62:42215)
0810 14:45:07.106775 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:07.106792 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:07.106829 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:45:07.107510 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:07.107531 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:07.107913 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:07.108258 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:07.108647 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:07.109893 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:07.111165 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:07.111748 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:07.111878 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:07.112756 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:07.112761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:07.112777 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:45:07.113239 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:07.113302 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:45:07.113327 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:07.113417 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:07.113581 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 14:45:07.113817 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":118,"threads_started":1}
I20260810 14:45:07.115469 17933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6e13a0cc694ff5b2231ae672dd55e9" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42215 } }
I20260810 14:45:07.115727 17933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:45:07.115790 17933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6e13a0cc694ff5b2231ae672dd55e9, State: Initialized, Role: FOLLOWER
I20260810 14:45:07.116257 17933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6e13a0cc694ff5b2231ae672dd55e9" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42215 } }
I20260810 14:45:07.116398 17933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:45:07.116514 17933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:45:07.116631 17933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:45:07.117755 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:45:07.117810 17933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6e13a0cc694ff5b2231ae672dd55e9" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42215 } }
I20260810 14:45:07.118170 17933 leader_election.cc:302] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c6e13a0cc694ff5b2231ae672dd55e9; no voters:
I20260810 14:45:07.118487 17933 leader_election.cc:288] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:45:07.118607 17942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:45:07.118861 17942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [term 1 LEADER]: Becoming Leader. State: Replica: 3c6e13a0cc694ff5b2231ae672dd55e9, State: Running, Role: LEADER
I20260810 14:45:07.119320 17942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6e13a0cc694ff5b2231ae672dd55e9" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42215 } }
I20260810 14:45:07.119881 17933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:45:07.121191 17946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c6e13a0cc694ff5b2231ae672dd55e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6e13a0cc694ff5b2231ae672dd55e9" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42215 } } }
I20260810 14:45:07.121338 17946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:45:07.121690 17944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c6e13a0cc694ff5b2231ae672dd55e9. Latest consensus state: current_term: 1 leader_uuid: "3c6e13a0cc694ff5b2231ae672dd55e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6e13a0cc694ff5b2231ae672dd55e9" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 42215 } } }
I20260810 14:45:07.121775 17944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9 [sys.catalog]: This master's current role is: LEADER
W20260810 14:45:07.125085 17966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:45:07.125262 17966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:45:07.125366 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:07.120060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59903 (local address 127.15.84.62:42215)
0810 14:45:07.120188 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:07.120192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:07.120627 (+ 435us) server_negotiation.cc:407] Connection header received
0810 14:45:07.120832 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:07.120836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:07.120884 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:07.120934 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:07.121498 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:07.122267 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:07.123493 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:07.123822 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:07.123929 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:07.124852 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:07.124856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:07.124859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:45:07.124874 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:07.124904 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:45:07.124918 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:07.124993 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:07.125114 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 14:45:07.125203 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:45:07.125388 17960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:45:07.126086 17960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:45:07.131927 17960 catalog_manager.cc:1347] Generated new cluster ID: fb3a7d4a978945dcb5c70b6b48889b72
I20260810 14:45:07.132014 17960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:45:07.150101 17960 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:45:07.151500 17960 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:45:07.163280 17960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c6e13a0cc694ff5b2231ae672dd55e9: Generated new TSK 0
I20260810 14:45:07.164234 17960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:45:07.171037 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 17821
2026-08-10T14:45:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1579 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:45:07 dist-test-slave-rmhg krb5kdc[17974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:45:07 dist-test-slave-rmhg krb5kdc[17974](info): set up 2 sockets
Aug 10 14:45:07 dist-test-slave-rmhg krb5kdc[17974](info): commencing operation
krb5kdc: starting...
W20260810 14:45:08.331451 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:45:08 dist-test-slave-rmhg krb5kdc[17974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:45:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:45:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:45:08.460676 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:46531
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:46531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:45:08.576939 17995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:45:08.577289 17995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:45:08.577337 17995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:45:08.581465 17995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:45:08.581552 17995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:45:08.581569 17995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:45:08.581584 17995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:45:08.581598 17995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:45:08.587299 17995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:46531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:46531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:45:08.588491 17995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:45:08.589615 17995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:45:08.598697 18005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:08.598847 18010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:08.598886 17995 server_base.cc:1034] running on GCE node
W20260810 14:45:08.598691 18006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:08.599591 17995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:45:08.600807 17995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:45:08.601974 17995 hybrid_clock.cc:648] HybridClock initialized: now 1786373108601954 us; error 36 us; skew 500 ppm
Aug 10 14:45:08 dist-test-slave-rmhg krb5kdc[17974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373108, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:45:08.604846 17995 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:45:08.605859 17995 webserver.cc:460] Webserver started at http://127.15.84.62:35133/ using document root <none> and password file <none>
I20260810 14:45:08.606410 17995 fs_manager.cc:359] Metadata directory not provided
I20260810 14:45:08.606469 17995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:45:08.606650 17995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:45:08.608752 17995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "8e13abc5e23b4966b8f03aeabaa93a8e"
format_stamp: "Formatted at 2026-08-10 14:45:08 on dist-test-slave-rmhg"
server_key: "2fdac37fd3ae377dc09a160001870b3d"
server_key_iv: "0bd3d4ea04663cfe675fb3c0e63cc31f"
server_key_version: "encryptionkey@0"
I20260810 14:45:08.609318 17995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "8e13abc5e23b4966b8f03aeabaa93a8e"
format_stamp: "Formatted at 2026-08-10 14:45:08 on dist-test-slave-rmhg"
server_key: "2fdac37fd3ae377dc09a160001870b3d"
server_key_iv: "0bd3d4ea04663cfe675fb3c0e63cc31f"
server_key_version: "encryptionkey@0"
I20260810 14:45:08.613078 17995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:45:08.615650 18019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:45:08.616950 17995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 14:45:08.617067 17995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "8e13abc5e23b4966b8f03aeabaa93a8e"
format_stamp: "Formatted at 2026-08-10 14:45:08 on dist-test-slave-rmhg"
server_key: "2fdac37fd3ae377dc09a160001870b3d"
server_key_iv: "0bd3d4ea04663cfe675fb3c0e63cc31f"
server_key_version: "encryptionkey@0"
I20260810 14:45:08.617198 17995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:45:08.621826 17995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:45:08.624920 17995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:45:08.625073 17995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:45:08.631950 17995 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:46531
I20260810 14:45:08.631997 18106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:46531 every 8 connection(s)
I20260810 14:45:08.632925 17995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:45:08.635847 18107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:45:08.638626 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 17995
I20260810 14:45:08.638778 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:45:08.639046 15696 external_mini_cluster.cc:1428] Setting key 05f0e955f9841d57eab03c2a2bad2117
I20260810 14:45:08.642180 18107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e: Bootstrap starting.
I20260810 14:45:08.644446 18111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:08.640451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47881 (local address 127.15.84.62:46531)
0810 14:45:08.640886 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:08.640903 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:08.640943 (+ 40us) server_negotiation.cc:407] Connection header received
0810 14:45:08.641632 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:08.641654 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:08.641977 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:08.642281 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:08.643158 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:08.643163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:08.643176 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:45:08.643189 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:08.643247 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:45:08.643269 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:08.643379 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:08.643497 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 14:45:08.643654 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":117,"threads_started":1}
I20260810 14:45:08.644940 18107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e: Neither blocks nor log segments found. Creating new log.
I20260810 14:45:08.645862 18107 log.cc:824] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:45:08.647717 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:45:08.648064 18107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e: No bootstrap required, opened a new log
I20260810 14:45:08.651608 18111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:08.649884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42485 (local address 127.15.84.62:46531)
0810 14:45:08.650019 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:08.650024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:08.650257 (+ 233us) server_negotiation.cc:407] Connection header received
0810 14:45:08.650354 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:08.650357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:08.650401 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:08.650482 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:08.651231 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:08.651236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:08.651239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:45:08.651253 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:08.651289 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:45:08.651304 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:08.651389 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:08.651442 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 14:45:08.651481 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:45:08.652274 18107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e13abc5e23b4966b8f03aeabaa93a8e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46531 } }
I20260810 14:45:08.652529 18107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:45:08.652590 18107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e13abc5e23b4966b8f03aeabaa93a8e, State: Initialized, Role: FOLLOWER
I20260810 14:45:08.653044 18107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e13abc5e23b4966b8f03aeabaa93a8e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46531 } }
I20260810 14:45:08.653213 18107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:45:08.653271 18107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:45:08.653348 18107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:45:08.654264 18107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e13abc5e23b4966b8f03aeabaa93a8e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46531 } }
I20260810 14:45:08.654588 18107 leader_election.cc:302] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e13abc5e23b4966b8f03aeabaa93a8e; no voters:
I20260810 14:45:08.654873 18107 leader_election.cc:288] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:45:08.655000 18120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:45:08.655262 18120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [term 1 LEADER]: Becoming Leader. State: Replica: 8e13abc5e23b4966b8f03aeabaa93a8e, State: Running, Role: LEADER
I20260810 14:45:08.655565 18120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e13abc5e23b4966b8f03aeabaa93a8e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46531 } }
I20260810 14:45:08.656104 18107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:45:08.657552 18124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e13abc5e23b4966b8f03aeabaa93a8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e13abc5e23b4966b8f03aeabaa93a8e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46531 } } }
I20260810 14:45:08.657578 18125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e13abc5e23b4966b8f03aeabaa93a8e. Latest consensus state: current_term: 1 leader_uuid: "8e13abc5e23b4966b8f03aeabaa93a8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e13abc5e23b4966b8f03aeabaa93a8e" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 46531 } } }
I20260810 14:45:08.657713 18125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [sys.catalog]: This master's current role is: LEADER
I20260810 14:45:08.657714 18124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e [sys.catalog]: This master's current role is: LEADER
W20260810 14:45:08.661266 18144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:45:08.661338 18144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:45:08.661424 18136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:45:08.662128 18136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:45:08.667407 18136 catalog_manager.cc:1347] Generated new cluster ID: 41d6b27b51bd4830b93c797d8a7b8c6f
I20260810 14:45:08.667490 18136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:45:08.685822 18136 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:45:08.686210 18136 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:45:08.690631 18136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e13abc5e23b4966b8f03aeabaa93a8e: Generated new TSK 0
I20260810 14:45:08.691448 18136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:45:08.707257 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 17995
2026-08-10T14:45:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1533 ms)
[----------] 18 tests from AuthTokenIssuingTest (28003 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:45:08 dist-test-slave-rmhg krb5kdc[18154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:45:08 dist-test-slave-rmhg krb5kdc[18154](info): set up 2 sockets
Aug 10 14:45:08 dist-test-slave-rmhg krb5kdc[18154](info): commencing operation
krb5kdc: starting...
W20260810 14:45:09.863466 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:45:09 dist-test-slave-rmhg krb5kdc[18154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:45:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:45:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:45:09.994688 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:38671
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:34631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:38671,127.15.84.61:42317,127.15.84.60:39461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:45:10.112125 18168 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:45:10.112444 18168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:45:10.112469 18168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:45:10.112507 18168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:45:10.116602 18168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:45:10.116695 18168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:45:10.116714 18168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:45:10.116729 18168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:45:10.116743 18168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:45:10.122356 18168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:38671,127.15.84.61:42317,127.15.84.60:39461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:38671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:45:10.123566 18168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:45:10.124675 18168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:45:10.133983 18174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:10.133968 18175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:10.133975 18179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:10.134366 18168 server_base.cc:1034] running on GCE node
I20260810 14:45:10.134968 18168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:45:10.136152 18168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:45:10.137310 18168 hybrid_clock.cc:648] HybridClock initialized: now 1786373110137292 us; error 35 us; skew 500 ppm
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:45:10.140295 18168 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:45:10.141373 18168 webserver.cc:460] Webserver started at http://127.15.84.62:38263/ using document root <none> and password file <none>
I20260810 14:45:10.141937 18168 fs_manager.cc:359] Metadata directory not provided
I20260810 14:45:10.141999 18168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:45:10.142189 18168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:45:10.144342 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "4c2537082fec4899aad96bd77a220243"
format_stamp: "Formatted at 2026-08-10 14:45:10 on dist-test-slave-rmhg"
server_key: "9c236697aba934a5a24f9260b330e3bf"
server_key_iv: "cb8b474a986f1541b01798029956df1e"
server_key_version: "encryptionkey@0"
I20260810 14:45:10.144918 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "4c2537082fec4899aad96bd77a220243"
format_stamp: "Formatted at 2026-08-10 14:45:10 on dist-test-slave-rmhg"
server_key: "9c236697aba934a5a24f9260b330e3bf"
server_key_iv: "cb8b474a986f1541b01798029956df1e"
server_key_version: "encryptionkey@0"
I20260810 14:45:10.149044 18168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:45:10.151837 18192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:45:10.153254 18168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:45:10.153406 18168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "4c2537082fec4899aad96bd77a220243"
format_stamp: "Formatted at 2026-08-10 14:45:10 on dist-test-slave-rmhg"
server_key: "9c236697aba934a5a24f9260b330e3bf"
server_key_iv: "cb8b474a986f1541b01798029956df1e"
server_key_version: "encryptionkey@0"
I20260810 14:45:10.153522 18168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:45:10.174671 18168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:45:10.178102 18168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:45:10.178320 18168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:45:10.185600 18168 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:38671
I20260810 14:45:10.185890 18267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:38671 every 8 connection(s)
I20260810 14:45:10.186676 18168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:45:10.189733 18271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:45:10.192736 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 18168
I20260810 14:45:10.192852 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:45:10.193118 15696 external_mini_cluster.cc:1428] Setting key b6094cbd81831e8f8865b84a991ac995
I20260810 14:45:10.193883 18271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.84.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.61@KRBTEST.COM" created.
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:45:10.212723 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.196357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:38671 (local address 127.0.0.1:53808)
0810 14:45:10.196912 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.196969 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:10.197969 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:10.198874 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:10.198879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:10.199261 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:10.199566 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.199584 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.201386 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.201389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.202269 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.202280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.203202 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.203205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.203307 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.203907 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:10.203930 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:10.207007 (+ 3077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:10.209351 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.209361 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.209372 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.209652 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.210062 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.210066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.210067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.210214 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.211044 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:10.211050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:10.211201 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 14:45:10.211361 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 14:45:10.211846 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":176,"thread_start_us":145,"threads_started":1}
I20260810 14:45:10.212728 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.196358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53808 (local address 127.15.84.62:38671)
0810 14:45:10.196956 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.196976 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.197099 (+ 123us) server_negotiation.cc:407] Connection header received
0810 14:45:10.198354 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.198361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.198654 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.198874 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.199742 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.201246 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.202406 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.203033 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.203188 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.207212 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.207219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.207233 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.207259 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.209228 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.209777 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.209784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.209786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.209935 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.210317 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.210320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.210321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.210911 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.211051 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.211484 (+ 433us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.211767 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":471,"spinlock_wait_cycles":19200,"thread_start_us":176,"threads_started":1}
I20260810 14:45:10.215278 18271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:10.216089 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.215821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:42317 (local address 127.0.0.1:41376)
0810 14:45:10.215952 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.216027 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:42317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 14:45:10.216305 18196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.84.61:42317: connect: Connection refused (error 111)
I20260810 14:45:10.217123 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.216902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:42317 (local address 127.0.0.1:41378)
0810 14:45:10.217013 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.217054 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:42317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 14:45:10.217314 18271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.61:42317: Network error: Client connection negotiation failed: client connection to 127.15.84.61:42317: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.61.keytab.
Entry for principal kudu/127.15.84.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.61.keytab.
I20260810 14:45:10.224660 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.61:42317
--webserver_interface=127.15.84.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.61.keytab
--principal=kudu/127.15.84.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:34631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:38671,127.15.84.61:42317,127.15.84.60:39461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:45:10.266588 18271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } attempt: 1
I20260810 14:45:10.267505 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.267191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:42317 (local address 127.0.0.1:41384)
0810 14:45:10.267342 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.267423 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:42317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 14:45:10.268311 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.268092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:42317 (local address 127.0.0.1:41394)
0810 14:45:10.268200 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.268247 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:42317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 14:45:10.268458 18271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.61:42317: Network error: Client connection negotiation failed: client connection to 127.15.84.61:42317: connect: Connection refused (error 111)
I20260810 14:45:10.336658 18271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } attempt: 2
I20260810 14:45:10.337566 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.337256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:42317 (local address 127.0.0.1:41408)
0810 14:45:10.337404 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.337476 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:42317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 14:45:10.338346 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.338122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:42317 (local address 127.0.0.1:41418)
0810 14:45:10.338236 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.338287 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:42317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 14:45:10.338495 18271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.61:42317: Network error: Client connection negotiation failed: client connection to 127.15.84.61:42317: connect: Connection refused (error 111)
W20260810 14:45:10.342553 18281 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:45:10.342875 18281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:45:10.342921 18281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:45:10.342983 18281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:45:10.347173 18281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:45:10.347249 18281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:45:10.347276 18281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:45:10.347298 18281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:45:10.347327 18281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:45:10.352916 18281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:38671,127.15.84.61:42317,127.15.84.60:39461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.61.keytab
--principal=kudu/127.15.84.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.61:42317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.84.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:45:10.354127 18281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:45:10.355175 18281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:45:10.363731 18291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:10.363757 18287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:10.363732 18289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:10.364076 18281 server_base.cc:1034] running on GCE node
I20260810 14:45:10.364712 18281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:45:10.365947 18281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:45:10.367112 18281 hybrid_clock.cc:648] HybridClock initialized: now 1786373110367083 us; error 40 us; skew 500 ppm
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:45:10.369892 18281 init.cc:377] Logged in from keytab as kudu/127.15.84.61@KRBTEST.COM (short username kudu)
I20260810 14:45:10.370802 18281 webserver.cc:460] Webserver started at http://127.15.84.61:39987/ using document root <none> and password file <none>
I20260810 14:45:10.371362 18281 fs_manager.cc:359] Metadata directory not provided
I20260810 14:45:10.371423 18281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:45:10.371651 18281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:45:10.373873 18281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/data/instance:
uuid: "fb31120573e2473e82c26558789faacf"
format_stamp: "Formatted at 2026-08-10 14:45:10 on dist-test-slave-rmhg"
server_key: "77fb135d3be5a7c41ffe4e825013d415"
server_key_iv: "193b348df7f2a82271a8dc2304c7012f"
server_key_version: "encryptionkey@0"
I20260810 14:45:10.374461 18281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/wal/instance:
uuid: "fb31120573e2473e82c26558789faacf"
format_stamp: "Formatted at 2026-08-10 14:45:10 on dist-test-slave-rmhg"
server_key: "77fb135d3be5a7c41ffe4e825013d415"
server_key_iv: "193b348df7f2a82271a8dc2304c7012f"
server_key_version: "encryptionkey@0"
I20260810 14:45:10.378470 18281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:45:10.381779 18300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:45:10.383617 18281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:45:10.383790 18281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/wal
uuid: "fb31120573e2473e82c26558789faacf"
format_stamp: "Formatted at 2026-08-10 14:45:10 on dist-test-slave-rmhg"
server_key: "77fb135d3be5a7c41ffe4e825013d415"
server_key_iv: "193b348df7f2a82271a8dc2304c7012f"
server_key_version: "encryptionkey@0"
I20260810 14:45:10.383926 18281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:45:10.398494 18281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:45:10.401700 18281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:45:10.401930 18281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:45:10.409464 18281 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.61:42317
I20260810 14:45:10.409739 18374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.61:42317 every 8 connection(s)
I20260810 14:45:10.410521 18281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/data/info.pb
I20260810 14:45:10.412297 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 18281
I20260810 14:45:10.412427 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-1/wal/instance
I20260810 14:45:10.412694 15696 external_mini_cluster.cc:1428] Setting key 5dd1397711cf8dee35d464a87a39fe3f
I20260810 14:45:10.413551 18375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:45:10.419948 18375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.84.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.60@KRBTEST.COM" created.
I20260810 14:45:10.429673 18271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } attempt: 3
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.61@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for kudu/127.15.84.61@KRBTEST.COM
I20260810 14:45:10.438717 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.423306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.15.84.62:38671)
0810 14:45:10.423491 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.423496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.424487 (+ 991us) server_negotiation.cc:407] Connection header received
0810 14:45:10.425661 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.425664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.425763 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.425840 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.427235 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.427787 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.429467 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.429762 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.429868 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.433963 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.433967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.433970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.433994 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.435758 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.436442 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.436445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.436447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.436539 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.436917 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.436920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.436922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.437109 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.437239 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.438418 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.438551 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:45:10.439299 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.423734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:38671 (local address 127.0.0.1:53824)
0810 14:45:10.424297 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.424349 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:10.425398 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:10.426059 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:10.426067 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:10.426516 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:10.427067 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.427081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.427935 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.427938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.429297 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.429312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.430008 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.430012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.430540 (+ 528us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.431718 (+ 1178us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:10.431744 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:10.433757 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:10.435888 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.435896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.435917 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.436330 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.436648 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.436650 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.436652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.436801 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.437239 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:10.437247 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:10.437560 (+ 313us) client_negotiation.cc:769] Sending connection context
0810 14:45:10.437742 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 14:45:10.438121 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":160,"threads_started":1}
I20260810 14:45:10.439587 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.430146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:42317 (local address 127.0.0.1:41434)
0810 14:45:10.430293 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.430309 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:10.430394 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:10.431103 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:10.431106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:10.431308 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:10.431433 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.431440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.432505 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.432509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.433088 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.433097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.433783 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.433787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.433897 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.434418 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:10.434431 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:10.435946 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:10.437945 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.437948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.437951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.438161 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.438490 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.438493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.438495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.438578 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.439103 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:10.439106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:10.439206 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 14:45:10.439312 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 14:45:10.439449 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 14:45:10.440420 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.430281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.15.84.61:42317)
0810 14:45:10.430639 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.430649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.430672 (+ 23us) server_negotiation.cc:407] Connection header received
0810 14:45:10.430710 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.430715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.430914 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.431096 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.431566 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.432391 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.433279 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.433657 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.433756 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.436078 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.436087 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.436100 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.436122 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.437826 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.438298 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.438305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.438306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.438392 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.438667 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.438670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.438672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.438984 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.439120 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.440097 (+ 977us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.440247 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":137,"threads_started":1}
I20260810 14:45:10.440881 18375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:10.442370 18271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:10.443077 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.442854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43826)
0810 14:45:10.442973 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.443022 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 14:45:10.443775 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.443519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43832)
0810 14:45:10.443662 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.443707 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260810 14:45:10.443912 18271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:39461: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.60.keytab.
Entry for principal kudu/127.15.84.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.60.keytab.
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.61@KRBTEST.COM for kudu/127.15.84.61@KRBTEST.COM
I20260810 14:45:10.446161 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.60:39461
--webserver_interface=127.15.84.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.60.keytab
--principal=kudu/127.15.84.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:34631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:38671,127.15.84.61:42317,127.15.84.60:39461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:45:10.449180 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.441462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.15.84.61:42317)
0810 14:45:10.441576 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.441580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.441593 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:45:10.441626 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.441628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.441668 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.441774 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.442139 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.442827 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.443778 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.444108 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.444198 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.446315 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.446318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.446320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.446341 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.447591 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.448005 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.448009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.448010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.448080 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.448331 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.448335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.448336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.448527 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.448612 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.448880 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.449025 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":50}
I20260810 14:45:10.449180 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.441384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:42317 (local address 127.0.0.1:41444)
0810 14:45:10.441477 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.441491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:10.441560 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:10.441795 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:10.441799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:10.441931 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:10.442030 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.442035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.442938 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.442941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.443643 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.443653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.444222 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.444225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.444310 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.444782 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:10.444797 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:10.446182 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:10.447709 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.447713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.447715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.447901 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.448169 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.448172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.448174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.448253 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.448617 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:10.448620 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:10.448714 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 14:45:10.448804 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 14:45:10.448965 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 14:45:10.450007 18375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:10.450771 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.450480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43840)
0810 14:45:10.450614 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.450704 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 14:45:10.450987 18301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
I20260810 14:45:10.451900 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.451687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43852)
0810 14:45:10.451799 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.451843 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 14:45:10.452055 18375 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:39461: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
I20260810 14:45:10.475051 18271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } attempt: 1
I20260810 14:45:10.475953 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.475648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43862)
0810 14:45:10.475807 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.475871 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 14:45:10.476737 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.476492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43872)
0810 14:45:10.476601 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.476671 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 14:45:10.476861 18271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:39461: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
I20260810 14:45:10.501338 18375 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } attempt: 1
I20260810 14:45:10.502300 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.501974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43874)
0810 14:45:10.502145 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.502214 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 14:45:10.503065 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.502819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43880)
0810 14:45:10.502930 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.502990 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 14:45:10.503223 18375 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:39461: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
I20260810 14:45:10.552062 18271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } attempt: 2
I20260810 14:45:10.552945 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.552627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43890)
0810 14:45:10.552797 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.552862 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 14:45:10.553830 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.553603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43900)
0810 14:45:10.553709 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.553766 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 14:45:10.553965 18271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:39461: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
W20260810 14:45:10.570583 18388 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:45:10.570899 18388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:45:10.570922 18388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:45:10.570967 18388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 14:45:10.571376 18375 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } attempt: 2
I20260810 14:45:10.572252 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.571912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43912)
0810 14:45:10.572087 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.572161 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 14:45:10.572950 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.572730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43928)
0810 14:45:10.572828 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.572880 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 14:45:10.573083 18375 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:39461: Network error: Client connection negotiation failed: client connection to 127.15.84.60:39461: connect: Connection refused (error 111)
W20260810 14:45:10.575662 18388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:45:10.575730 18388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:45:10.575750 18388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:45:10.575780 18388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:45:10.575810 18388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:45:10.581565 18388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:34631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:38671,127.15.84.61:42317,127.15.84.60:39461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.60.keytab
--principal=kudu/127.15.84.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.60:39461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.84.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:45:10.582787 18388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:45:10.583904 18388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:45:10.593348 18398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:10.593408 18397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:10.593461 18388 server_base.cc:1034] running on GCE node
W20260810 14:45:10.593375 18402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:10.594389 18388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:45:10.595677 18388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:45:10.596885 18388 hybrid_clock.cc:648] HybridClock initialized: now 1786373110596865 us; error 35 us; skew 500 ppm
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:45:10.600116 18388 init.cc:377] Logged in from keytab as kudu/127.15.84.60@KRBTEST.COM (short username kudu)
I20260810 14:45:10.601183 18388 webserver.cc:460] Webserver started at http://127.15.84.60:42733/ using document root <none> and password file <none>
I20260810 14:45:10.601811 18388 fs_manager.cc:359] Metadata directory not provided
I20260810 14:45:10.601879 18388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:45:10.602103 18388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:45:10.604349 18388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/data/instance:
uuid: "3edec8cff05f41c5bcbc7e333f0751b6"
format_stamp: "Formatted at 2026-08-10 14:45:10 on dist-test-slave-rmhg"
server_key: "e26f04562e9c33e50d4a98d72f4b2095"
server_key_iv: "e85d4382ef46cda74d67b84ef28e7cde"
server_key_version: "encryptionkey@0"
I20260810 14:45:10.604981 18388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/wal/instance:
uuid: "3edec8cff05f41c5bcbc7e333f0751b6"
format_stamp: "Formatted at 2026-08-10 14:45:10 on dist-test-slave-rmhg"
server_key: "e26f04562e9c33e50d4a98d72f4b2095"
server_key_iv: "e85d4382ef46cda74d67b84ef28e7cde"
server_key_version: "encryptionkey@0"
I20260810 14:45:10.609030 18388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 14:45:10.611713 18418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:45:10.613118 18388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:45:10.613334 18388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/wal
uuid: "3edec8cff05f41c5bcbc7e333f0751b6"
format_stamp: "Formatted at 2026-08-10 14:45:10 on dist-test-slave-rmhg"
server_key: "e26f04562e9c33e50d4a98d72f4b2095"
server_key_iv: "e85d4382ef46cda74d67b84ef28e7cde"
server_key_version: "encryptionkey@0"
I20260810 14:45:10.613467 18388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:45:10.627705 18388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:45:10.631151 18388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:45:10.631343 18388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:45:10.638638 18388 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.60:39461
I20260810 14:45:10.638944 18507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.60:39461 every 8 connection(s)
I20260810 14:45:10.639695 18388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/data/info.pb
I20260810 14:45:10.642730 18508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:45:10.645016 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 18388
I20260810 14:45:10.645203 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372819317171-15696-0/minicluster-data/master-2/wal/instance
I20260810 14:45:10.645537 15696 external_mini_cluster.cc:1428] Setting key c8452e7c04b619cf2760b2fd05610abf
I20260810 14:45:10.647574 18508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:10.653134 18271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } attempt: 3
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.60@KRBTEST.COM
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.60@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for kudu/127.15.84.60@KRBTEST.COM
I20260810 14:45:10.660548 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.649442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.15.84.62:38671)
0810 14:45:10.649584 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.649600 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.650052 (+ 452us) server_negotiation.cc:407] Connection header received
0810 14:45:10.650166 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.650168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.650212 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.650276 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.651199 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.651787 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.652978 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.653258 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.653357 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.656952 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.656957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.656960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.656987 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.658516 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.659104 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.659108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.659110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.659193 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.659496 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.659499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.659501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.659735 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.659828 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.660266 (+ 438us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.660384 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:45:10.661871 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.647099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.15.84.60:39461)
0810 14:45:10.647643 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.647661 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.647700 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:45:10.648441 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.648462 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.648838 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.649224 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.650236 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.651526 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.652453 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.653119 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.653330 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.656022 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.656030 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.656046 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.656074 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.658110 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.658611 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.658618 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.658625 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.658744 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.659019 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.659023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.659025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.659579 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.659791 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.660220 (+ 429us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.660540 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":348,"thread_start_us":167,"threads_started":1}
I20260810 14:45:10.661871 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.649372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:38671 (local address 127.0.0.1:53832)
0810 14:45:10.649884 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.649943 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:10.650080 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:10.650345 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:10.650351 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:10.650846 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:10.651052 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.651065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.651911 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.651914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.652851 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.652863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.653609 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.653614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.653724 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.654981 (+ 1257us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:10.655002 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:10.656784 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:10.658656 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.658667 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.658674 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.658963 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.659298 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.659302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.659304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.659399 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.659850 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:10.659858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:10.660013 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 14:45:10.660800 (+ 787us) client_negotiation.cc:240] Negotiation successful
0810 14:45:10.661000 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":64,"thread_start_us":143,"threads_started":1}
I20260810 14:45:10.663972 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.653993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43948)
0810 14:45:10.654129 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.654144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:10.654210 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:10.655299 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:10.655303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:10.655490 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:10.655612 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.655621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.656517 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.656521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.657210 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.657222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.658036 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.658040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.658143 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.658528 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:10.658541 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:10.659826 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:10.662196 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.662200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.662202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.662656 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.662980 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.662983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.662985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.663094 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.663482 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:10.663485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:10.663579 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 14:45:10.663670 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 14:45:10.663811 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 14:45:10.663903 18508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:10.664088 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.653747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43948 (local address 127.15.84.60:39461)
0810 14:45:10.655037 (+ 1290us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.655047 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.655060 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:45:10.655106 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.655109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.655151 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.655248 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.655762 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.656407 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.657355 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.657766 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.657901 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.659945 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.659949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.659952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.659975 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.662038 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.662786 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.662790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.662791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.662882 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.663187 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.663191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.663193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.663382 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.663471 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.663747 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.663856 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1217,"thread_start_us":109,"threads_started":1}
I20260810 14:45:10.664340 18375 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } attempt: 3
I20260810 14:45:10.667955 18271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243: Bootstrap starting.
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.60@KRBTEST.COM for kudu/127.15.84.61@KRBTEST.COM
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.61@KRBTEST.COM for kudu/127.15.84.60@KRBTEST.COM
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:45:10.670994 18271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243: Neither blocks nor log segments found. Creating new log.
I20260810 14:45:10.671974 18271 log.cc:824] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243: Log is configured to *not* fsync() on all Append() calls
I20260810 14:45:10.674585 18271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243: No bootstrap required, opened a new log
I20260810 14:45:10.675026 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.664866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43960)
0810 14:45:10.665529 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.665543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:10.665614 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:10.665936 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:10.665938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:10.666126 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:10.666219 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.666226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.667374 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.667377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.668121 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.668131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.668716 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.668720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.668810 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.669182 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:10.669197 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:10.671274 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:10.673272 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.673276 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.673278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.673527 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.673854 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.673857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.673859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.673949 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.674436 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:10.674438 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:10.674545 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 14:45:10.674703 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 14:45:10.674860 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595}
I20260810 14:45:10.675422 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.665067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.15.84.62:38671)
0810 14:45:10.665228 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.665232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.665244 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:45:10.665276 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.665279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.665318 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.665387 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.666213 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.666945 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.668043 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.668342 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.668434 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.671578 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.671582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.671585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.671613 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.673269 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.673758 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.673761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.673762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.673847 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.674140 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.674143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.674145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.674333 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.674495 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.675164 (+ 669us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.675287 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:45:10.675556 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.665410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.15.84.60:39461)
0810 14:45:10.665744 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.665748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.665758 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:45:10.665786 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.665788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.665823 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.666323 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.666401 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.667237 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.668251 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.668574 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.669007 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.671415 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.671420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.671423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.671449 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.673110 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.673656 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.673659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.673661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.673748 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.674059 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.674062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.674064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.674317 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.675233 (+ 916us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.675306 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.675413 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":269,"spinlock_wait_cycles":2432}
I20260810 14:45:10.676028 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.664666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.15.84.61:42317)
0810 14:45:10.665037 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.665041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.665054 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:45:10.665102 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.665105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.665181 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.665249 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.665718 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.666432 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.667476 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.667824 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.667898 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.672211 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.672215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.672218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.672246 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.674133 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.674661 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.674665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.674667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.674782 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.675094 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.675098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.675100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.675313 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.675376 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.675760 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.675881 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304}
I20260810 14:45:10.678931 18375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf: Bootstrap starting.
I20260810 14:45:10.680277 18271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } }
I20260810 14:45:10.680555 18271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2537082fec4899aad96bd77a220243, State: Initialized, Role: FOLLOWER
I20260810 14:45:10.680922 18271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } }
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.61@KRBTEST.COM
I20260810 14:45:10.681576 18517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } } }
I20260810 14:45:10.681823 18517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:45:10.682024 18375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf: Neither blocks nor log segments found. Creating new log.
I20260810 14:45:10.682493 18271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:45:10.683008 18375 log.cc:824] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf: Log is configured to *not* fsync() on all Append() calls
I20260810 14:45:10.683892 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.664451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:42317 (local address 127.0.0.1:41446)
0810 14:45:10.664598 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.664613 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:10.664690 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:10.665271 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:10.665273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:10.665450 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:10.665580 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.665587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.666550 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.666553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.667369 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.667378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.668147 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.668153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.668268 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.669176 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:10.669202 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:10.671673 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:10.674273 (+ 2600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.674276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.674278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.674516 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.674898 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.674901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.674903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.674986 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.675445 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:10.675447 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:10.675546 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 14:45:10.683465 (+ 7919us) client_negotiation.cc:240] Negotiation successful
0810 14:45:10.683705 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":19,"spinlock_wait_cycles":896}
I20260810 14:45:10.684789 18508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:10.685379 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.676825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.15.84.61:42317)
0810 14:45:10.676929 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.676932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.676943 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:45:10.676975 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.676977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.677012 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.677073 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.677714 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.678203 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.678952 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.679229 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.679320 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.681985 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.681989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.681992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.682022 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.683561 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.684041 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.684045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.684047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.684131 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.684444 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.684447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.684449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.684692 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.684863 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.685077 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.685240 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 14:45:10.687391 18375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf: No bootstrap required, opened a new log
W20260810 14:45:10.688946 18536 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:45:10.689077 18536 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:45:10.690939 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Aug 10 14:45:10 dist-test-slave-rmhg krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.60@KRBTEST.COM for kudu/127.15.84.60@KRBTEST.COM
I20260810 14:45:10.691700 18375 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } }
I20260810 14:45:10.691953 18375 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb31120573e2473e82c26558789faacf, State: Initialized, Role: FOLLOWER
I20260810 14:45:10.692296 18375 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } }
I20260810 14:45:10.692934 18230 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4c2537082fec4899aad96bd77a220243"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53842
I20260810 14:45:10.693269 18230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 14:45:10.693380 18230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:45:10.693892 18375 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:45:10.690162 18168 master_runner.cc:186] Error getting master registration for 127.15.84.61:42317: OK, instance_id { permanent_uuid: "fb31120573e2473e82c26558789faacf" instance_seqno: 1786373110407415 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:45:10.694029 18168 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:45:10.694679 18230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } }
I20260810 14:45:10.694932 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.685343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:39461 (local address 127.0.0.1:43962)
0810 14:45:10.685857 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:10.685870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:10.685979 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:10.686264 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:10.686266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:10.686436 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:10.686522 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.686528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.687478 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.687481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.688306 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:10.688320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.689023 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.689026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:10.689117 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.689998 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:10.690013 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:10.691561 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:10.693414 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.693417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.693419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.693647 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.693913 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:10.693916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:10.693917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:10.693999 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:10.694435 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:10.694438 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:10.694516 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 14:45:10.694605 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 14:45:10.694754 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":91,"threads_started":1}
I20260810 14:45:10.695498 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.685668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.15.84.60:39461)
0810 14:45:10.685778 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.685782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.685918 (+ 136us) server_negotiation.cc:407] Connection header received
0810 14:45:10.686061 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.686063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.686108 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.686149 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.686637 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.687355 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.688431 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.688775 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.688851 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.691696 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.691699 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.691701 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.691725 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.693306 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.693746 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.693749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.693751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.693830 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.694085 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.694098 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.694100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.694314 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.694403 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.695248 (+ 845us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.695362 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:45:10.696314 18230 leader_election.cc:288] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [CANDIDATE]: Term 1 election: Requested vote from peers fb31120573e2473e82c26558789faacf (127.15.84.61:42317), 3edec8cff05f41c5bcbc7e333f0751b6 (127.15.84.60:39461)
I20260810 14:45:10.698575 18508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6: Bootstrap starting.
I20260810 14:45:10.701025 18508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6: Neither blocks nor log segments found. Creating new log.
I20260810 14:45:10.702049 18508 log.cc:824] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6: Log is configured to *not* fsync() on all Append() calls
I20260810 14:45:10.701419 18541 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } } }
I20260810 14:45:10.701591 18335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c2537082fec4899aad96bd77a220243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fb31120573e2473e82c26558789faacf"
I20260810 14:45:10.704668 18335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf [term 0 FOLLOWER]: Advancing to term 1
W20260810 14:45:10.705546 18558 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:45:10.705631 18558 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:45:10.706509 18335 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c2537082fec4899aad96bd77a220243 in term 1.
I20260810 14:45:10.707667 18196 leader_election.cc:302] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c2537082fec4899aad96bd77a220243, fb31120573e2473e82c26558789faacf; no voters:
I20260810 14:45:10.707988 18517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:45:10.708263 18517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [term 1 LEADER]: Becoming Leader. State: Replica: 4c2537082fec4899aad96bd77a220243, State: Running, Role: LEADER
I20260810 14:45:10.708621 18517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } }
I20260810 14:45:10.710045 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.698575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.15.84.62:38671)
0810 14:45:10.698689 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.698694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.699587 (+ 893us) server_negotiation.cc:407] Connection header received
0810 14:45:10.699865 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.699869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.699920 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.699972 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.701715 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.702407 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.703616 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.703946 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.704061 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.706408 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.706412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.706415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.706444 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.708127 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.708601 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.708605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.708607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.708728 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.709010 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.709013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.709015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.709257 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.709367 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.709764 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.709889 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 14:45:10.710615 18517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c2537082fec4899aad96bd77a220243. Latest consensus state: current_term: 1 leader_uuid: "4c2537082fec4899aad96bd77a220243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } } }
I20260810 14:45:10.710719 18517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [sys.catalog]: This master's current role is: LEADER
I20260810 14:45:10.711290 18578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:45:10.712623 18578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:45:10.708969 18281 master_runner.cc:186] Error getting master registration for 127.15.84.60:39461: OK, instance_id { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" instance_seqno: 1786373110636584 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:45:10.713858 18281 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:45:10.713948 18541 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:45:10.714924 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.699054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.15.84.60:39461)
0810 14:45:10.699172 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.699176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.705982 (+ 6806us) server_negotiation.cc:407] Connection header received
0810 14:45:10.706080 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.706083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.706175 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.706259 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.707006 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.707705 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.708877 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.709245 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.709325 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.710839 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.710842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.710845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.710872 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.712472 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.712967 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.712971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.712973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.713066 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.713617 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.713621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.713623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.713844 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.713965 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.714643 (+ 678us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.714771 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:45:10.720436 18335 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf [term 1 FOLLOWER]: Refusing update from remote peer 4c2537082fec4899aad96bd77a220243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:45:10.721200 18517 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:45:10.725235 18541 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c2537082fec4899aad96bd77a220243. Latest consensus state: current_term: 1 leader_uuid: "4c2537082fec4899aad96bd77a220243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } } }
I20260810 14:45:10.725391 18541 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:45:10.727963 18508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6: No bootstrap required, opened a new log
I20260810 14:45:10.730451 18517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c2537082fec4899aad96bd77a220243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } } }
I20260810 14:45:10.730633 18517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [sys.catalog]: This master's current role is: LEADER
I20260810 14:45:10.732350 18541 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c2537082fec4899aad96bd77a220243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } } }
I20260810 14:45:10.732487 18541 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:45:10.732919 18579 mvcc.cc:205] Tried to move back new op lower bound from 7316984261500506112 to 7316984261466714112. Current Snapshot: MvccSnapshot[applied={T|T < 7316984261500506112}]
I20260810 14:45:10.733415 18580 mvcc.cc:205] Tried to move back new op lower bound from 7316984261500506112 to 7316984261466714112. Current Snapshot: MvccSnapshot[applied={T|T < 7316984261500506112}]
I20260810 14:45:10.737023 18578 catalog_manager.cc:1347] Generated new cluster ID: 156db8a5253b4be4a760ac95aa2416d0
I20260810 14:45:10.737107 18578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:45:10.748170 18508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } }
I20260810 14:45:10.750900 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:10.721712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.15.84.61:42317)
0810 14:45:10.721834 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.721839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.722142 (+ 303us) server_negotiation.cc:407] Connection header received
0810 14:45:10.722473 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.722477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.722530 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.722582 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.724334 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.725200 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.735682 (+ 10482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:10.736047 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:10.736128 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:10.742033 (+ 5905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:10.742038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:10.742041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:10.742074 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:10.743777 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.747401 (+ 3624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.747406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.747408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.747523 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:10.748569 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:10.748573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:10.748575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:10.748837 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:10.748970 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:10.750563 (+ 1593us) server_negotiation.cc:299] Negotiation successful
0810 14:45:10.750728 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:45:10.750381 18508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3edec8cff05f41c5bcbc7e333f0751b6, State: Initialized, Role: FOLLOWER
I20260810 14:45:10.752890 18508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } }
I20260810 14:45:10.755990 18578 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:45:10.757441 18578 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:45:10.760778 18460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c2537082fec4899aad96bd77a220243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3edec8cff05f41c5bcbc7e333f0751b6"
I20260810 14:45:10.761097 18460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:45:10.763154 18460 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c2537082fec4899aad96bd77a220243 in term 1.
I20260810 14:45:10.763733 18459 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6 [term 1 FOLLOWER]: Refusing update from remote peer 4c2537082fec4899aad96bd77a220243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:45:10.764792 18517 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:45:10.770663 18508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:45:10.775381 18575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c2537082fec4899aad96bd77a220243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } } }
I20260810 14:45:10.775527 18575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243 [sys.catalog]: This master's current role is: LEADER
I20260810 14:45:10.776309 18587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c2537082fec4899aad96bd77a220243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } } }
I20260810 14:45:10.776441 18587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:45:10.776671 18578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c2537082fec4899aad96bd77a220243: Generated new TSK 0
I20260810 14:45:10.776851 18587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c2537082fec4899aad96bd77a220243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } } }
I20260810 14:45:10.776983 18587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:45:10.777704 18578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:45:10.773015 18595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c2537082fec4899aad96bd77a220243. Latest consensus state: current_term: 1 leader_uuid: "4c2537082fec4899aad96bd77a220243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2537082fec4899aad96bd77a220243" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 38671 } } peers { permanent_uuid: "fb31120573e2473e82c26558789faacf" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 42317 } } peers { permanent_uuid: "3edec8cff05f41c5bcbc7e333f0751b6" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 39461 } } }
I20260810 14:45:10.778038 18595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 14:45:10.781232 18616 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:45:10.781315 18616 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:45:10.783169 18591 mvcc.cc:205] Tried to move back new op lower bound from 7316984261635817472 to 7316984261466714112. Current Snapshot: MvccSnapshot[applied={T|T < 7316984261500506112}]
W20260810 14:45:10.811751 18276 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:10.808866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.15.84.62:38671)
0810 14:45:10.809010 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.809015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.810481 (+ 1466us) server_negotiation.cc:407] Connection header received
0810 14:45:10.810726 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.810729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.810784 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.810837 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.811581 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53858: BlockingRecv error: recv got EOF from 127.0.0.1:53858 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 14:45:10.813485 18512 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:10.809781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.15.84.60:39461)
0810 14:45:10.809903 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.809908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.812470 (+ 2562us) server_negotiation.cc:407] Connection header received
0810 14:45:10.812670 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.812674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.812728 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.812778 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.813351 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43978: BlockingRecv error: recv got EOF from 127.0.0.1:43978 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 14:45:10.815901 18387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:10.814436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.15.84.61:42317)
0810 14:45:10.814554 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.814558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.814958 (+ 400us) server_negotiation.cc:407] Connection header received
0810 14:45:10.815129 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.815133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.815185 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.815235 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.815769 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41458: BlockingRecv error: recv got EOF from 127.0.0.1:41458 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 14:45:10.820971 18387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:10.819326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.15.84.61:42317)
0810 14:45:10.819436 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.819440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.819919 (+ 479us) server_negotiation.cc:407] Connection header received
0810 14:45:10.820157 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.820160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.820244 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.820297 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.820858 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41468: BlockingRecv error: recv got EOF from 127.0.0.1:41468 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 14:45:10.826478 18387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:10.824910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41472 (local address 127.15.84.61:42317)
0810 14:45:10.825011 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.825014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.825553 (+ 539us) server_negotiation.cc:407] Connection header received
0810 14:45:10.825754 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.825758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.825808 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.825857 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.826369 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41472: BlockingRecv error: recv got EOF from 127.0.0.1:41472 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 14:45:10.832454 18387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:10.830879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.15.84.61:42317)
0810 14:45:10.830987 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.830991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.831510 (+ 519us) server_negotiation.cc:407] Connection header received
0810 14:45:10.831719 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.831723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.831774 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.831824 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.832346 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41474: BlockingRecv error: recv got EOF from 127.0.0.1:41474 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 14:45:10.840487 18387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:10.838921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.15.84.61:42317)
0810 14:45:10.839060 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.839064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.839569 (+ 505us) server_negotiation.cc:407] Connection header received
0810 14:45:10.839767 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.839770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.839822 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.839870 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.840376 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41488: BlockingRecv error: recv got EOF from 127.0.0.1:41488 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 14:45:10.852999 18387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:10.851253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.15.84.61:42317)
0810 14:45:10.851372 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.851376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.851958 (+ 582us) server_negotiation.cc:407] Connection header received
0810 14:45:10.852182 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.852186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.852244 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.852298 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.852886 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41496: BlockingRecv error: recv got EOF from 127.0.0.1:41496 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 14:45:10.874606 18387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:10.872383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.15.84.61:42317)
0810 14:45:10.872522 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.872526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.873417 (+ 891us) server_negotiation.cc:407] Connection header received
0810 14:45:10.873692 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.873697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.873763 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.873824 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.874476 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41502: BlockingRecv error: recv got EOF from 127.0.0.1:41502 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 14:45:10.910792 18387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:10.909460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.15.84.61:42317)
0810 14:45:10.909589 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.909593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.909879 (+ 286us) server_negotiation.cc:407] Connection header received
0810 14:45:10.910021 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.910025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.910077 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.910170 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.910666 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41510: BlockingRecv error: recv got EOF from 127.0.0.1:41510 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 14:45:10.978473 18387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:10.977283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.15.84.61:42317)
0810 14:45:10.977470 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:10.977474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:10.977664 (+ 190us) server_negotiation.cc:407] Connection header received
0810 14:45:10.977800 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:10.977803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:10.977853 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:10.977930 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:10.978348 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41512: BlockingRecv error: recv got EOF from 127.0.0.1:41512 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260810 14:45:11.109886 18387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:11.108859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41526 (local address 127.15.84.61:42317)
0810 14:45:11.108995 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:11.108998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:11.109095 (+ 97us) server_negotiation.cc:407] Connection header received
0810 14:45:11.109257 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:11.109260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:11.109309 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:11.109394 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:11.109784 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41526: BlockingRecv error: recv got EOF from 127.0.0.1:41526 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 14:45:11.369396 18387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:11.368285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.15.84.61:42317)
0810 14:45:11.368433 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:11.368437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:11.368454 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:45:11.368632 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:11.368635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:11.368685 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:11.368756 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:11.369294 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41530: BlockingRecv error: recv got EOF from 127.0.0.1:41530 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:45:11.707044 18558 catalog_manager.cc:1259] Loaded cluster ID: 156db8a5253b4be4a760ac95aa2416d0
I20260810 14:45:11.707175 18558 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf: loading cluster ID for follower catalog manager: success
I20260810 14:45:11.708961 18558 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf: acquiring CA information for follower catalog manager: success
I20260810 14:45:11.710355 18558 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb31120573e2473e82c26558789faacf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:45:11.782775 18616 catalog_manager.cc:1259] Loaded cluster ID: 156db8a5253b4be4a760ac95aa2416d0
I20260810 14:45:11.782862 18616 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6: loading cluster ID for follower catalog manager: success
I20260810 14:45:11.784801 18616 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6: acquiring CA information for follower catalog manager: success
I20260810 14:45:11.786115 18616 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3edec8cff05f41c5bcbc7e333f0751b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:45:11.887941 18720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:11.883843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.15.84.61:42317)
0810 14:45:11.884131 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:11.884134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:11.884151 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:45:11.884260 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:11.884262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:11.884395 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:11.884514 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:45:11.884740 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:11.885342 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:11.886353 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:11.886605 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:11.886698 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:11.886995 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:45:11.887506 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:45:11.887650 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:11.887719 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 14:45:11.887777 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":105,"threads_started":1}
I20260810 14:45:11.895309 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:11.891324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.15.84.60:39461)
0810 14:45:11.891609 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:11.891613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:11.891629 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:45:11.891720 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:11.891723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:11.891847 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:11.891947 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:45:11.892145 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:11.892686 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:11.893778 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:11.894017 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:11.894100 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:11.894381 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:45:11.894910 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:45:11.895006 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:11.895086 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 14:45:11.895141 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":73,"threads_started":1}
I20260810 14:45:11.902974 18720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:11.898621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41546 (local address 127.15.84.61:42317)
0810 14:45:11.898759 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:11.898763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:11.899210 (+ 447us) server_negotiation.cc:407] Connection header received
0810 14:45:11.899316 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:11.899320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:11.899378 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:11.899452 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:45:11.899736 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:11.900562 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:11.901726 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:11.901979 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:11.902091 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:11.902382 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:45:11.902480 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:45:11.902553 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:11.902754 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 14:45:11.902824 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:45:11.903147 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:11.899029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.15.84.60:39461)
0810 14:45:11.899149 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:11.899166 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:11.899334 (+ 168us) server_negotiation.cc:407] Connection header received
0810 14:45:11.899429 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:11.899432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:11.899487 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:11.899598 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:45:11.899877 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:11.900619 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:11.901972 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:11.902264 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:11.902372 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:11.902650 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:45:11.902730 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:45:11.902794 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:11.902938 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 14:45:11.903011 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 14:45:11.905084 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:11.898596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.15.84.62:38671)
0810 14:45:11.898861 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:11.898865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:11.899431 (+ 566us) server_negotiation.cc:407] Connection header received
0810 14:45:11.899544 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:11.899547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:11.899735 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:11.899872 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:45:11.900985 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:11.901849 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:11.903443 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:11.903779 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:11.903876 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:11.904378 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:45:11.904704 (+ 326us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:45:11.904777 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:11.904897 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 14:45:11.904957 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260810 14:45:11.908525 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 18168
I20260810 14:45:11.928030 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 18281
I20260810 14:45:11.943676 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 18388
2026-08-10T14:45:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3245 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:45:12 dist-test-slave-rmhg krb5kdc[18747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:45:12 dist-test-slave-rmhg krb5kdc[18747](info): set up 2 sockets
Aug 10 14:45:12 dist-test-slave-rmhg krb5kdc[18747](info): commencing operation
krb5kdc: starting...
W20260810 14:45:13.098127 15696 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/test-user.keytab.
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:45:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
Entry for principal kudu/127.15.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab.
I20260810 14:45:13.229931 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:37257
--webserver_interface=127.15.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:44029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:37257,127.15.84.61:46423,127.15.84.60:41059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:45:13.345769 18773 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:45:13.346097 18773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:45:13.346135 18773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:45:13.346199 18773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:45:13.350324 18773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:45:13.350404 18773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:45:13.350430 18773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:45:13.350450 18773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:45:13.350471 18773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:45:13.355983 18773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:44029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:37257,127.15.84.61:46423,127.15.84.60:41059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.62.keytab
--principal=kudu/127.15.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.62:37257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:45:13.357110 18773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:45:13.358186 18773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:45:13.366942 18773 server_base.cc:1034] running on GCE node
W20260810 14:45:13.366889 18781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:13.366889 18782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:13.366883 18787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:13.367794 18773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:45:13.368973 18773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:45:13.370186 18773 hybrid_clock.cc:648] HybridClock initialized: now 1786373113370167 us; error 58 us; skew 500 ppm
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:45:13.373106 18773 init.cc:377] Logged in from keytab as kudu/127.15.84.62@KRBTEST.COM (short username kudu)
I20260810 14:45:13.374125 18773 webserver.cc:460] Webserver started at http://127.15.84.62:35801/ using document root <none> and password file <none>
I20260810 14:45:13.374725 18773 fs_manager.cc:359] Metadata directory not provided
I20260810 14:45:13.374799 18773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:45:13.375044 18773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:45:13.377291 18773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/data/instance:
uuid: "6fa8ea6e07194e1697763b6e23ad4959"
format_stamp: "Formatted at 2026-08-10 14:45:13 on dist-test-slave-rmhg"
server_key: "54b60e577395cc9a97652a4bafd589d3"
server_key_iv: "370ca383e66eb441ef7396dd768d4b6c"
server_key_version: "encryptionkey@0"
I20260810 14:45:13.377848 18773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/wal/instance:
uuid: "6fa8ea6e07194e1697763b6e23ad4959"
format_stamp: "Formatted at 2026-08-10 14:45:13 on dist-test-slave-rmhg"
server_key: "54b60e577395cc9a97652a4bafd589d3"
server_key_iv: "370ca383e66eb441ef7396dd768d4b6c"
server_key_version: "encryptionkey@0"
I20260810 14:45:13.381680 18773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:45:13.384183 18801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:45:13.385612 18773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:45:13.385751 18773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/wal
uuid: "6fa8ea6e07194e1697763b6e23ad4959"
format_stamp: "Formatted at 2026-08-10 14:45:13 on dist-test-slave-rmhg"
server_key: "54b60e577395cc9a97652a4bafd589d3"
server_key_iv: "370ca383e66eb441ef7396dd768d4b6c"
server_key_version: "encryptionkey@0"
I20260810 14:45:13.385854 18773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:45:13.406229 18773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:45:13.409260 18773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:45:13.409433 18773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:45:13.416579 18773 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.62:37257
I20260810 14:45:13.416862 18873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.62:37257 every 8 connection(s)
I20260810 14:45:13.417662 18773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/data/info.pb
I20260810 14:45:13.420603 18874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:45:13.424525 18874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:13.427863 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 18773
I20260810 14:45:13.427983 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-0/wal/instance
I20260810 14:45:13.428275 15696 external_mini_cluster.cc:1428] Setting key 7e9c247d59bfe6b0bd4f006185ffa3f9
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.61@KRBTEST.COM" created.
I20260810 14:45:13.440563 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.426877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53862 (local address 127.15.84.62:37257)
0810 14:45:13.427330 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.427346 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.427576 (+ 230us) server_negotiation.cc:407] Connection header received
0810 14:45:13.428767 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.428774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.429028 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.429252 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.430094 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.431095 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.432423 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.432798 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.432918 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.435751 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.435757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.435768 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.435790 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.437479 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.437961 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.437966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.437967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.438089 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.438367 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.438369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.438371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.438781 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.438889 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.439335 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.439633 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":305,"thread_start_us":130,"threads_started":1}
I20260810 14:45:13.440567 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.426877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37257 (local address 127.0.0.1:53862)
0810 14:45:13.427413 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.427464 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:13.428415 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:13.429300 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:13.429305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:13.429667 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:13.429945 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.429959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.431234 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.431238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.432317 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.432328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.432970 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.432976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.433114 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.433736 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:13.433759 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:13.435579 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:13.437619 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.437631 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.437642 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.437875 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.438159 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.438161 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.438162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.438266 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.438913 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:13.438918 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:13.439069 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 14:45:13.439228 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 14:45:13.439720 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":177,"thread_start_us":166,"threads_started":1}
I20260810 14:45:13.443149 18874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:13.443994 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.443709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:46423 (local address 127.0.0.1:51004)
0810 14:45:13.443841 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.443923 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 14:45:13.444209 18804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.84.61:46423: connect: Connection refused (error 111)
I20260810 14:45:13.445087 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.444843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:46423 (local address 127.0.0.1:51010)
0810 14:45:13.444963 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.445018 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 14:45:13.445288 18874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.61:46423: Network error: Client connection negotiation failed: client connection to 127.15.84.61:46423: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.61.keytab.
Entry for principal kudu/127.15.84.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.61.keytab.
I20260810 14:45:13.456173 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.61:46423
--webserver_interface=127.15.84.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.61.keytab
--principal=kudu/127.15.84.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:44029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:37257,127.15.84.61:46423,127.15.84.60:41059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:45:13.494553 18874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } attempt: 1
I20260810 14:45:13.495494 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.495182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:46423 (local address 127.0.0.1:51022)
0810 14:45:13.495345 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.495412 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 14:45:13.496232 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.496008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:46423 (local address 127.0.0.1:51024)
0810 14:45:13.496115 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.496167 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 14:45:13.496354 18874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.61:46423: Network error: Client connection negotiation failed: client connection to 127.15.84.61:46423: connect: Connection refused (error 111)
I20260810 14:45:13.564548 18874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } attempt: 2
I20260810 14:45:13.565451 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.565115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:46423 (local address 127.0.0.1:51034)
0810 14:45:13.565308 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.565373 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 14:45:13.566148 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.565942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:46423 (local address 127.0.0.1:51042)
0810 14:45:13.566045 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.566091 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.61:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 14:45:13.566259 18874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.61:46423: Network error: Client connection negotiation failed: client connection to 127.15.84.61:46423: connect: Connection refused (error 111)
W20260810 14:45:13.574887 18881 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:45:13.575204 18881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:45:13.575250 18881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:45:13.575314 18881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:45:13.579484 18881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:45:13.579564 18881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:45:13.579592 18881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:45:13.579614 18881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:45:13.579643 18881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:45:13.585145 18881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:44029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:37257,127.15.84.61:46423,127.15.84.60:41059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.61.keytab
--principal=kudu/127.15.84.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.61:46423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.84.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:45:13.586340 18881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:45:13.587400 18881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:45:13.596271 18888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:13.596233 18890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:13.596374 18881 server_base.cc:1034] running on GCE node
W20260810 14:45:13.596277 18895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:13.597218 18881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:45:13.598405 18881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:45:13.599571 18881 hybrid_clock.cc:648] HybridClock initialized: now 1786373113599547 us; error 38 us; skew 500 ppm
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:45:13.606709 18881 init.cc:377] Logged in from keytab as kudu/127.15.84.61@KRBTEST.COM (short username kudu)
I20260810 14:45:13.607729 18881 webserver.cc:460] Webserver started at http://127.15.84.61:41899/ using document root <none> and password file <none>
I20260810 14:45:13.608418 18881 fs_manager.cc:359] Metadata directory not provided
I20260810 14:45:13.608525 18881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:45:13.608807 18881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:45:13.611253 18881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/data/instance:
uuid: "805fc62f4e2e407bae8798f417f50c4a"
format_stamp: "Formatted at 2026-08-10 14:45:13 on dist-test-slave-rmhg"
server_key: "6ebaf4d394bb29347ee0bb142e0d708f"
server_key_iv: "ada0b7315cdaff52488b27eb9542c5a3"
server_key_version: "encryptionkey@0"
I20260810 14:45:13.611789 18881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/wal/instance:
uuid: "805fc62f4e2e407bae8798f417f50c4a"
format_stamp: "Formatted at 2026-08-10 14:45:13 on dist-test-slave-rmhg"
server_key: "6ebaf4d394bb29347ee0bb142e0d708f"
server_key_iv: "ada0b7315cdaff52488b27eb9542c5a3"
server_key_version: "encryptionkey@0"
I20260810 14:45:13.615638 18881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:45:13.618135 18904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:45:13.619395 18881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:45:13.619504 18881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/wal
uuid: "805fc62f4e2e407bae8798f417f50c4a"
format_stamp: "Formatted at 2026-08-10 14:45:13 on dist-test-slave-rmhg"
server_key: "6ebaf4d394bb29347ee0bb142e0d708f"
server_key_iv: "ada0b7315cdaff52488b27eb9542c5a3"
server_key_version: "encryptionkey@0"
I20260810 14:45:13.619619 18881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:45:13.633711 18881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:45:13.636755 18881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:45:13.636924 18881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:45:13.644263 18881 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.61:46423
I20260810 14:45:13.644526 18988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.61:46423 every 8 connection(s)
I20260810 14:45:13.645367 18881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/data/info.pb
I20260810 14:45:13.648272 18991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:45:13.652200 18991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:13.654162 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 18881
I20260810 14:45:13.654286 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-1/wal/instance
I20260810 14:45:13.654554 15696 external_mini_cluster.cc:1428] Setting key 4490def9be91031e54ca913e04275aa5
I20260810 14:45:13.657405 18874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } attempt: 3
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.61@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for kudu/127.15.84.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.84.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.84.60@KRBTEST.COM" created.
I20260810 14:45:13.668143 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.654165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.15.84.62:37257)
0810 14:45:13.654332 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.654337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.655302 (+ 965us) server_negotiation.cc:407] Connection header received
0810 14:45:13.656531 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.656534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.656626 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.656728 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.658212 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.658760 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.660209 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.660460 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.660553 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.663545 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.663550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.663553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.663583 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.665581 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.666228 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.666231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.666232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.666305 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.666785 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.666788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.666790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.667034 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.667122 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.667808 (+ 686us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.667958 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:45:13.668283 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.657858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:46423 (local address 127.0.0.1:51046)
0810 14:45:13.658061 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.658075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:13.658151 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:13.658859 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:13.658862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:13.659065 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:13.659193 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.659200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.660659 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.660663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.661500 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.661513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.662296 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.662300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.662407 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.662815 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:13.662830 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:13.664408 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:13.666629 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.666632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.666633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.666876 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.667218 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.667221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.667223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.667310 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.667803 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:13.667806 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:13.667895 (+ 89us) client_negotiation.cc:769] Sending connection context
0810 14:45:13.668005 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 14:45:13.668128 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260810 14:45:13.669514 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.657969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.15.84.61:46423)
0810 14:45:13.658392 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.658403 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.658427 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:45:13.658466 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.658471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.658687 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.658846 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.659318 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.660500 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.661631 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.662148 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.662287 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.664612 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.664621 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.664638 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.664665 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.666503 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.667006 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.667012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.667014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.667107 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.667418 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.667421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.667423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.667683 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.668559 (+ 876us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.668661 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.668782 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":324,"thread_start_us":126,"threads_started":1}
I20260810 14:45:13.669531 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.654561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37257 (local address 127.0.0.1:53866)
0810 14:45:13.655112 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.655174 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:13.656280 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:13.656905 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:13.656912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:13.657334 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:13.658022 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.658046 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.658924 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.658928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.660036 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.660050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.660627 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.660633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.660873 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.661481 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:13.661497 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:13.663383 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:13.665701 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.665713 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.665725 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.666135 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.666421 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.666424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.666426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.666651 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.667185 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:13.667191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:13.667492 (+ 301us) client_negotiation.cc:769] Sending connection context
0810 14:45:13.667732 (+ 240us) client_negotiation.cc:240] Negotiation successful
0810 14:45:13.668086 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":38,"thread_start_us":127,"threads_started":1}
I20260810 14:45:13.671316 18991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:13.672097 18874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:13.672731 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.672508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44220)
0810 14:45:13.672613 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.672666 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 14:45:13.673446 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.673225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44224)
0810 14:45:13.673351 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.673397 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 14:45:13.673600 18874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:41059: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.61@KRBTEST.COM for kudu/127.15.84.61@KRBTEST.COM
I20260810 14:45:13.679997 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.671995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51054 (local address 127.15.84.61:46423)
0810 14:45:13.672107 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.672111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.672124 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:45:13.672158 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.672161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.672199 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.672268 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.672815 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.673617 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.674652 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.674893 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.674971 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.677032 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.677035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.677037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.677057 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.678471 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.678868 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.678872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.678874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.678945 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.679181 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.679185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.679187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.679358 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.679476 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.679751 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.679855 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:45:13.679998 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.671876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:46423 (local address 127.0.0.1:51054)
0810 14:45:13.671988 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.672001 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:13.672078 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:13.672380 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:13.672383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:13.672569 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:13.672697 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.672704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.673733 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.673736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.674530 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.674542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.675001 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.675005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.675114 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.675536 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:13.675551 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:13.676904 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:13.678570 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.678573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.678576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.678789 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.679023 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.679026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.679028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.679106 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.679444 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:13.679447 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:13.679537 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 14:45:13.679667 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 14:45:13.679807 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 14:45:13.680821 18991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:13.681496 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.681272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44228)
0810 14:45:13.681370 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.681444 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 14:45:13.682046 18906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
I20260810 14:45:13.682893 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.682709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44240)
0810 14:45:13.682804 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.682847 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 14:45:13.683048 18991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:41059: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.84.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.60.keytab.
Entry for principal kudu/127.15.84.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.60.keytab.
I20260810 14:45:13.686255 15696 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
/tmp/dist-test-taskPiaUUw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.60:41059
--webserver_interface=127.15.84.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.60.keytab
--principal=kudu/127.15.84.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.84.20:44029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.84.62:37257,127.15.84.61:46423,127.15.84.60:41059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:45:13.704782 18874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } attempt: 1
I20260810 14:45:13.705740 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.705404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44248)
0810 14:45:13.705588 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.705654 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 14:45:13.706465 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.706242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44264)
0810 14:45:13.706354 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.706399 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 14:45:13.706636 18874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:41059: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
I20260810 14:45:13.732311 18991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } attempt: 1
I20260810 14:45:13.733352 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.732978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44270)
0810 14:45:13.733175 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.733250 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 14:45:13.734205 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.733933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44284)
0810 14:45:13.734072 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.734127 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 14:45:13.734404 18991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:41059: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
I20260810 14:45:13.781785 18874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } attempt: 2
I20260810 14:45:13.782845 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.782452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44290)
0810 14:45:13.782647 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.782730 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 14:45:13.783645 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.783420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44306)
0810 14:45:13.783527 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.783585 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 14:45:13.783859 18874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:41059: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
I20260810 14:45:13.802564 18991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } attempt: 2
I20260810 14:45:13.803392 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.803098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44312)
0810 14:45:13.803243 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.803309 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 14:45:13.804194 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.803976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44322)
0810 14:45:13.804073 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.804133 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 14:45:13.804322 18991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.84.60:41059: Network error: Client connection negotiation failed: client connection to 127.15.84.60:41059: connect: Connection refused (error 111)
W20260810 14:45:13.817317 18998 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:45:13.817644 18998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:45:13.817680 18998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:45:13.817735 18998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:45:13.821838 18998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:45:13.821913 18998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:45:13.821931 18998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:45:13.821947 18998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:45:13.821961 18998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:45:13.827752 18998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.84.20:44029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.84.62:37257,127.15.84.61:46423,127.15.84.60:41059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/krb5kdc/kudu_127.15.84.60.keytab
--principal=kudu/127.15.84.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.84.60:41059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.84.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:45:13.828893 18998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:45:13.829948 18998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:45:13.838891 18998 server_base.cc:1034] running on GCE node
W20260810 14:45:13.838845 19010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:13.838848 19013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:45:13.838852 19011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:45:13.839781 18998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:45:13.840984 18998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:45:13.842166 18998 hybrid_clock.cc:648] HybridClock initialized: now 1786373113842145 us; error 38 us; skew 500 ppm
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:45:13.844918 18998 init.cc:377] Logged in from keytab as kudu/127.15.84.60@KRBTEST.COM (short username kudu)
I20260810 14:45:13.845929 18998 webserver.cc:460] Webserver started at http://127.15.84.60:39537/ using document root <none> and password file <none>
I20260810 14:45:13.846495 18998 fs_manager.cc:359] Metadata directory not provided
I20260810 14:45:13.846563 18998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:45:13.846782 18998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:45:13.848879 18998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/data/instance:
uuid: "50c08fbdc26848c4a26844ea8f5ae0b8"
format_stamp: "Formatted at 2026-08-10 14:45:13 on dist-test-slave-rmhg"
server_key: "74a9bd41e7919ca7ee51510276852aee"
server_key_iv: "b5dc5ea8a836da44f559cdb957dcb436"
server_key_version: "encryptionkey@0"
I20260810 14:45:13.849459 18998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/wal/instance:
uuid: "50c08fbdc26848c4a26844ea8f5ae0b8"
format_stamp: "Formatted at 2026-08-10 14:45:13 on dist-test-slave-rmhg"
server_key: "74a9bd41e7919ca7ee51510276852aee"
server_key_iv: "b5dc5ea8a836da44f559cdb957dcb436"
server_key_version: "encryptionkey@0"
I20260810 14:45:13.853240 18998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:45:13.855679 19023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:45:13.856946 18998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:45:13.857071 18998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/data,/tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/wal
uuid: "50c08fbdc26848c4a26844ea8f5ae0b8"
format_stamp: "Formatted at 2026-08-10 14:45:13 on dist-test-slave-rmhg"
server_key: "74a9bd41e7919ca7ee51510276852aee"
server_key_iv: "b5dc5ea8a836da44f559cdb957dcb436"
server_key_version: "encryptionkey@0"
I20260810 14:45:13.857234 18998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:45:13.869959 18998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:45:13.872833 18998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:45:13.873023 18998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:45:13.880234 18998 rpc_server.cc:307] RPC server started. Bound to: 127.15.84.60:41059
I20260810 14:45:13.880501 19098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.84.60:41059 every 8 connection(s)
I20260810 14:45:13.881309 18998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/data/info.pb
I20260810 14:45:13.883018 18874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } attempt: 3
I20260810 14:45:13.884615 19099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:45:13.885927 15696 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu as pid 18998
I20260810 14:45:13.886047 15696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPiaUUw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372819317171-15696-0/minicluster-data/master-2/wal/instance
I20260810 14:45:13.886319 15696 external_mini_cluster.cc:1428] Setting key 5e83976bcdbbb68dc47b7b285caf00c4
I20260810 14:45:13.889631 19099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.62@KRBTEST.COM for kudu/127.15.84.60@KRBTEST.COM
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.60@KRBTEST.COM
I20260810 14:45:13.895462 18991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } attempt: 3
I20260810 14:45:13.897614 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.883499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44334)
0810 14:45:13.883633 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.883649 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:13.883781 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:13.886548 (+ 2767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:13.886552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:13.886768 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:13.886907 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.886917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.888753 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.888757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.889609 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.889623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.890445 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.890449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.890539 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.891072 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:13.891087 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:13.892651 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:13.895168 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.895171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.895173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.895417 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.895888 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.895891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.895893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.895991 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.897080 (+ 1089us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:13.897084 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:13.897211 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 14:45:13.897302 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 14:45:13.897431 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.60@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:45:13.899392 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.884280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44334 (local address 127.15.84.60:41059)
0810 14:45:13.884731 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.884753 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.884799 (+ 46us) server_negotiation.cc:407] Connection header received
0810 14:45:13.885708 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.885733 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.886195 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.886586 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.887051 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.888606 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.889729 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.890286 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.890492 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.892876 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.892883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.892900 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.892930 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.895041 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.895548 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.895553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.895559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.895782 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.896105 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.896107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.896110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.896954 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.898214 (+ 1260us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.898289 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.898396 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":274,"server-negotiator.queue_time_us":284,"spinlock_wait_cycles":6912,"thread_start_us":124,"threads_started":1}
I20260810 14:45:13.899394 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.887353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.15.84.60:41059)
0810 14:45:13.887657 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.887665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.887677 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:45:13.887706 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.887708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.887748 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.887826 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.888604 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.889426 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.890645 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.890980 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.891079 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.893863 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.893867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.893869 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.893895 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.895371 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.896124 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.896127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.896129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.896215 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.896544 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.896547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.896548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.896957 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.897784 (+ 827us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.897972 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.898306 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":226,"thread_start_us":101,"threads_started":1}
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.61@KRBTEST.COM for kudu/127.15.84.60@KRBTEST.COM
I20260810 14:45:13.904496 18874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959: Bootstrap starting.
I20260810 14:45:13.904896 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.891895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.62:37257 (local address 127.0.0.1:53868)
0810 14:45:13.892329 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.892379 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:13.892535 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:13.892915 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:13.892921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:13.893515 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:13.893659 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.893668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.894612 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.894615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.895540 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.895547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.896170 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.896174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.896336 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.896802 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:13.896815 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:13.899135 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:13.902256 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.902266 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.902271 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.902639 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.903310 (+ 671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.903314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.903316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.903413 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.903946 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:13.903952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:13.904072 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 14:45:13.904523 (+ 451us) client_negotiation.cc:240] Negotiation successful
0810 14:45:13.904719 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":3,"thread_start_us":168,"threads_started":1}
I20260810 14:45:13.905668 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.895911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44352)
0810 14:45:13.896124 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.896140 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:13.896217 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:13.897172 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:13.897176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:13.897354 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:13.897461 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.897468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.898551 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.898555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.899315 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.899327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.900143 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.900146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.900230 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.900562 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:13.900575 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:13.902066 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:13.904042 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.904045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.904047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.904298 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.904643 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.904646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.904648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.904741 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.905210 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:13.905214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:13.905317 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 14:45:13.905395 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:45:13.905524 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260810 14:45:13.906323 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.896405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.15.84.60:41059)
0810 14:45:13.896888 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.896897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.896910 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:45:13.896942 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.896945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.896996 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.897571 (+ 575us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.897681 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.898428 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.899463 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.900035 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.900123 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.902211 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.902215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.902217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.902242 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.903916 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.904422 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.904425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.904427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.904529 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.904852 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.904855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.904857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.905053 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.905966 (+ 913us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.906045 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.906171 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":104,"threads_started":1}
I20260810 14:45:13.907418 18874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959: Neither blocks nor log segments found. Creating new log.
I20260810 14:45:13.908252 18874 log.cc:824] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959: Log is configured to *not* fsync() on all Append() calls
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.62@KRBTEST.COM
I20260810 14:45:13.909333 18991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a: Bootstrap starting.
I20260810 14:45:13.909380 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.891912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.15.84.62:37257)
0810 14:45:13.892021 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.892025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.892450 (+ 425us) server_negotiation.cc:407] Connection header received
0810 14:45:13.892667 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.892670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.892720 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.892767 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.893791 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.894491 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.895648 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.895956 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.896056 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.899288 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.899292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.899294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.899320 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.902145 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.903086 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.903089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.903091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.903193 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.903507 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.903510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.903512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.903783 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.903860 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.904308 (+ 448us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.904423 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:45:13.910384 19099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:13.910935 18874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959: No bootstrap required, opened a new log
I20260810 14:45:13.912451 18991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a: Neither blocks nor log segments found. Creating new log.
I20260810 14:45:13.913285 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.902906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.15.84.62:37257)
0810 14:45:13.903226 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.903241 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.903252 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:45:13.903282 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.903284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.903328 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.904163 (+ 835us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.904254 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.905034 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.905894 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.906123 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.906207 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.908940 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.908944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.908946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.908972 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.910953 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.911484 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.911487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.911489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.911570 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.911876 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.911879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.911881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.912082 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.912875 (+ 793us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.912953 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.913070 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":91,"threads_started":1}
I20260810 14:45:13.913380 18991 log.cc:824] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:45:13.914404 18874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } }
I20260810 14:45:13.914647 18874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa8ea6e07194e1697763b6e23ad4959, State: Initialized, Role: FOLLOWER
I20260810 14:45:13.914969 18874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } }
I20260810 14:45:13.916368 18874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:45:13.916731 18991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a: No bootstrap required, opened a new log
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.60@KRBTEST.COM for kudu/127.15.84.61@KRBTEST.COM
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.84.61@KRBTEST.COM
I20260810 14:45:13.921195 18991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } }
I20260810 14:45:13.921440 18991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 805fc62f4e2e407bae8798f417f50c4a, State: Initialized, Role: FOLLOWER
I20260810 14:45:13.921819 18991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } }
I20260810 14:45:13.922379 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.910894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.61:46423 (local address 127.0.0.1:51066)
0810 14:45:13.913305 (+ 2411us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.913320 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:13.913559 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:13.913851 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:13.913855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:13.914030 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:13.914161 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.914168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.915167 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.915170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.915941 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.915953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.916621 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.916624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.916724 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.917209 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:13.917223 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:13.918701 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:13.920602 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.920606 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.920608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.920847 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.921207 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.921209 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.921211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.921305 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.921832 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:13.921835 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:13.921941 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 14:45:13.922068 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 14:45:13.922222 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2327}
I20260810 14:45:13.922407 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.910985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.15.84.61:46423)
0810 14:45:13.911178 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.911183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.913487 (+ 2304us) server_negotiation.cc:407] Connection header received
0810 14:45:13.913686 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.913690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.913742 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.913825 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.914295 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.915020 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.916078 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.916380 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.916458 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.918861 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.918865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.918867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.918891 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.920468 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.920964 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.920967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.920969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.921056 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.921410 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.921414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.921416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.921691 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.921757 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.922161 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.922274 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260810 14:45:13.923116 19099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:45:13.923276 18991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:45:13.922765 18773 master_runner.cc:186] Error getting master registration for 127.15.84.61:46423: OK, instance_id { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" instance_seqno: 1786373113642176 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:45:13.926716 18773 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:45:13.926748 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.914639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.15.84.61:46423)
0810 14:45:13.914902 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.914909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.915233 (+ 324us) server_negotiation.cc:407] Connection header received
0810 14:45:13.915272 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.915274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.915319 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.915397 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.916208 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.917083 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.918130 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.918577 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.918673 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.921614 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.921618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.921620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.921647 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.924252 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.924767 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.924771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.924773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.924875 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.925265 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.925269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.925270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.925482 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.925664 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.926372 (+ 708us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.926488 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
I20260810 14:45:13.927294 19135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } } }
I20260810 14:45:13.927420 19135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:45:13.929700 19117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } } }
I20260810 14:45:13.929833 19117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [sys.catalog]: This master's current role is: FOLLOWER
Aug 10 14:45:13 dist-test-slave-rmhg krb5kdc[18747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373113, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.84.60@KRBTEST.COM for kudu/127.15.84.60@KRBTEST.COM
I20260810 14:45:13.930367 15696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:45:13.931576 18838 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6fa8ea6e07194e1697763b6e23ad4959"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53870
I20260810 14:45:13.931787 18838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 14:45:13.931891 18838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [term 0 FOLLOWER]: Advancing to term 1
W20260810 14:45:13.932832 19157 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:45:13.932901 19157 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:45:13.933264 18838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } }
I20260810 14:45:13.933570 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.923870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.15.84.60:41059)
0810 14:45:13.925304 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.925308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.925321 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:45:13.925353 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.925356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.925397 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.925470 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.925996 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.926750 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.927716 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.928017 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.928099 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.930371 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.930374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.930377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.930396 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.931992 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.932422 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.932425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.932427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.932491 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.932724 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.932727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.932729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.932913 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.932980 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.933297 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.933416 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1359}
I20260810 14:45:13.934005 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.923554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.84.60:41059 (local address 127.0.0.1:44364)
0810 14:45:13.923987 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 14:45:13.924001 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:45:13.924134 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:45:13.925617 (+ 1483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:45:13.925620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:45:13.925781 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:45:13.925882 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.925889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.926842 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.926845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.927597 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:45:13.927609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.928227 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.928230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:45:13.928303 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.928765 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:45:13.928780 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:45:13.930243 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:45:13.932119 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.932122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.932124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.932335 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.932576 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:45:13.932578 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:45:13.932580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:45:13.932647 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:45:13.933033 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:45:13.933036 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:45:13.933121 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 14:45:13.933735 (+ 614us) client_negotiation.cc:240] Negotiation successful
0810 14:45:13.933855 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":111,"threads_started":1}
I20260810 14:45:13.930960 18881 master_runner.cc:186] Error getting master registration for 127.15.84.60:41059: OK, instance_id { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" instance_seqno: 1786373113878262 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:45:13.934311 18881 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:45:13.935230 18838 leader_election.cc:288] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [CANDIDATE]: Term 1 election: Requested vote from peers 805fc62f4e2e407bae8798f417f50c4a (127.15.84.61:46423), 50c08fbdc26848c4a26844ea8f5ae0b8 (127.15.84.60:41059)
I20260810 14:45:13.935530 18948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fa8ea6e07194e1697763b6e23ad4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "805fc62f4e2e407bae8798f417f50c4a"
I20260810 14:45:13.935768 18948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:45:13.937042 18948 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa8ea6e07194e1697763b6e23ad4959 in term 1.
I20260810 14:45:13.937585 19099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8: Bootstrap starting.
I20260810 14:45:13.937727 18804 leader_election.cc:302] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa8ea6e07194e1697763b6e23ad4959, 805fc62f4e2e407bae8798f417f50c4a; no voters:
I20260810 14:45:13.938074 19117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:45:13.938413 19117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [term 1 LEADER]: Becoming Leader. State: Replica: 6fa8ea6e07194e1697763b6e23ad4959, State: Running, Role: LEADER
I20260810 14:45:13.939002 19117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } }
I20260810 14:45:13.940490 19099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8: Neither blocks nor log segments found. Creating new log.
I20260810 14:45:13.941455 19099 log.cc:824] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8: Log is configured to *not* fsync() on all Append() calls
I20260810 14:45:13.942296 19117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fa8ea6e07194e1697763b6e23ad4959. Latest consensus state: current_term: 1 leader_uuid: "6fa8ea6e07194e1697763b6e23ad4959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } } }
I20260810 14:45:13.942494 19117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [sys.catalog]: This master's current role is: LEADER
W20260810 14:45:13.944001 19134 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:45:13.944068 19134 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:45:13.945909 19099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8: No bootstrap required, opened a new log
I20260810 14:45:13.947580 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.937389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.15.84.62:37257)
0810 14:45:13.937497 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.937502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.937649 (+ 147us) server_negotiation.cc:407] Connection header received
0810 14:45:13.937959 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.937962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.938013 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.938061 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.939182 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.939956 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.941295 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.941662 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.941747 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.943748 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.943752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.943754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.943783 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.945648 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.946187 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.946190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.946192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.946301 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.946587 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.946590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.946592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.946840 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.946971 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.947313 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.947434 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:45:13.953459 19176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:45:13.954525 19176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:45:13.959321 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.949566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.15.84.60:41059)
0810 14:45:13.949677 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.949681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.949694 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:45:13.949803 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.949806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.949852 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.949920 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.950875 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.951646 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.952766 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.953089 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.953193 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.954943 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.954947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.954949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.954980 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.956865 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.957379 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.957383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.957384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.957478 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.957792 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.957795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.957797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.958261 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.958401 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.959029 (+ 628us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.959150 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:45:13.961668 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:13.949951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51082 (local address 127.15.84.61:46423)
0810 14:45:13.950078 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:13.950083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:13.950441 (+ 358us) server_negotiation.cc:407] Connection header received
0810 14:45:13.950727 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:13.950730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:13.950788 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:13.950841 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:13.952220 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.952979 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.954951 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:13.955329 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:13.955417 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:13.957737 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:45:13.957741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:45:13.957744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:45:13.957775 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:45:13.959359 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.960149 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.960153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.960155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.960252 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:45:13.960572 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:45:13.960576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:45:13.960578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:45:13.960817 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:45:13.960942 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:13.961376 (+ 434us) server_negotiation.cc:299] Negotiation successful
0810 14:45:13.961500 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:45:13.962719 19099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } }
I20260810 14:45:13.963006 19099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c08fbdc26848c4a26844ea8f5ae0b8, State: Initialized, Role: FOLLOWER
I20260810 14:45:13.963371 19099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } }
I20260810 14:45:13.967972 18948 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a [term 1 FOLLOWER]: Refusing update from remote peer 6fa8ea6e07194e1697763b6e23ad4959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:45:13.967948 19180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } } }
I20260810 14:45:13.968092 19180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:45:13.969357 19099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:45:13.969627 19059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:45:13.970718 19172 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:45:13.971191 19059 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8 [term 1 FOLLOWER]: Refusing update from remote peer 6fa8ea6e07194e1697763b6e23ad4959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:45:13.971658 19061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fa8ea6e07194e1697763b6e23ad4959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8"
I20260810 14:45:13.971793 19117 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:45:13.972970 19061 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa8ea6e07194e1697763b6e23ad4959 in term 1.
I20260810 14:45:13.976153 19180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fa8ea6e07194e1697763b6e23ad4959. Latest consensus state: current_term: 1 leader_uuid: "6fa8ea6e07194e1697763b6e23ad4959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } } }
I20260810 14:45:13.976284 19180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:45:13.976576 19135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fa8ea6e07194e1697763b6e23ad4959. Latest consensus state: current_term: 1 leader_uuid: "6fa8ea6e07194e1697763b6e23ad4959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } } }
I20260810 14:45:13.976684 19135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a [sys.catalog]: This master's current role is: FOLLOWER
W20260810 14:45:13.988031 19202 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 14:45:13.992720 19172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fa8ea6e07194e1697763b6e23ad4959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } } }
I20260810 14:45:13.992905 19172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [sys.catalog]: This master's current role is: LEADER
W20260810 14:45:13.994936 19202 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:45:13.995466 19135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fa8ea6e07194e1697763b6e23ad4959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } } }
I20260810 14:45:13.995587 19135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:45:13.995997 19172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fa8ea6e07194e1697763b6e23ad4959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } } }
I20260810 14:45:13.996124 19172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959 [sys.catalog]: This master's current role is: LEADER
I20260810 14:45:13.996654 19179 mvcc.cc:205] Tried to move back new op lower bound from 7316984274803859456 to 7316984274700697600. Current Snapshot: MvccSnapshot[applied={T|T < 7316984274803859456}]
I20260810 14:45:13.997049 19180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fa8ea6e07194e1697763b6e23ad4959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa8ea6e07194e1697763b6e23ad4959" member_type: VOTER last_known_addr { host: "127.15.84.62" port: 37257 } } peers { permanent_uuid: "805fc62f4e2e407bae8798f417f50c4a" member_type: VOTER last_known_addr { host: "127.15.84.61" port: 46423 } } peers { permanent_uuid: "50c08fbdc26848c4a26844ea8f5ae0b8" member_type: VOTER last_known_addr { host: "127.15.84.60" port: 41059 } } }
I20260810 14:45:13.997417 19180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:45:13.997885 19195 mvcc.cc:205] Tried to move back new op lower bound from 7316984274803859456 to 7316984274700697600. Current Snapshot: MvccSnapshot[applied={T|T < 7316984274803859456}]
I20260810 14:45:14.000142 19176 catalog_manager.cc:1347] Generated new cluster ID: aa3b5e5f95bc4789ac8f221d526e0014
I20260810 14:45:14.000216 19176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:45:14.021726 19176 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:45:14.029366 19176 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:45:14.062922 19176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fa8ea6e07194e1697763b6e23ad4959: Generated new TSK 0
I20260810 14:45:14.063817 19176 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:45:14.105050 19115 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:14.101317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51084 (local address 127.15.84.61:46423)
0810 14:45:14.101482 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:14.101486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:14.103784 (+ 2298us) server_negotiation.cc:407] Connection header received
0810 14:45:14.104012 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:14.104016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:14.104072 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:14.104139 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:14.104889 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51084: BlockingRecv error: recv got EOF from 127.0.0.1:51084 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 14:45:14.106765 19109 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:14.102405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.15.84.60:41059)
0810 14:45:14.102556 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:14.102561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:14.105682 (+ 3121us) server_negotiation.cc:407] Connection header received
0810 14:45:14.105892 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:14.105895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:14.105954 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:14.106011 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:14.106621 (+ 610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44382: BlockingRecv error: recv got EOF from 127.0.0.1:44382 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 14:45:14.108209 19111 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:14.103172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.15.84.62:37257)
0810 14:45:14.103307 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:14.103311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:14.107219 (+ 3908us) server_negotiation.cc:407] Connection header received
0810 14:45:14.107370 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:14.107373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:14.107430 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:14.107509 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:14.108078 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53892: BlockingRecv error: recv got EOF from 127.0.0.1:53892 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 14:45:14.112679 19115 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:14.111191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.15.84.61:46423)
0810 14:45:14.111302 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:14.111307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:14.111747 (+ 440us) server_negotiation.cc:407] Connection header received
0810 14:45:14.111955 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:14.111958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:14.112036 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:14.112085 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:14.112573 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51090: BlockingRecv error: recv got EOF from 127.0.0.1:51090 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 14:45:14.118014 19115 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:14.116354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.15.84.61:46423)
0810 14:45:14.116464 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:14.116468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:14.116974 (+ 506us) server_negotiation.cc:407] Connection header received
0810 14:45:14.117261 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:14.117264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:14.117320 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:14.117371 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:14.117898 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51102: BlockingRecv error: recv got EOF from 127.0.0.1:51102 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 14:45:14.124505 19115 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:14.122519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.15.84.61:46423)
0810 14:45:14.122634 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:14.122639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:14.123245 (+ 606us) server_negotiation.cc:407] Connection header received
0810 14:45:14.123513 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:14.123517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:14.123582 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:14.123661 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:14.124351 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51118: BlockingRecv error: recv got EOF from 127.0.0.1:51118 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 14:45:14.132808 19115 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:14.130989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.15.84.61:46423)
0810 14:45:14.131163 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:14.131169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:14.131787 (+ 618us) server_negotiation.cc:407] Connection header received
0810 14:45:14.132015 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:14.132018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:14.132074 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:14.132141 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:14.132702 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51128: BlockingRecv error: recv got EOF from 127.0.0.1:51128 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260810 14:45:14.145237 19115 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:14.143936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51130 (local address 127.15.84.61:46423)
0810 14:45:14.144058 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:14.144062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:14.144331 (+ 269us) server_negotiation.cc:407] Connection header received
0810 14:45:14.144487 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:14.144491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:14.144544 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:14.144625 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:14.145080 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51130: BlockingRecv error: recv got EOF from 127.0.0.1:51130 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 14:45:14.164789 19115 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:14.163440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.15.84.61:46423)
0810 14:45:14.163575 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:14.163579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:14.163780 (+ 201us) server_negotiation.cc:407] Connection header received
0810 14:45:14.163927 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:14.163930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:14.164010 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:14.164092 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:14.164662 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51146: BlockingRecv error: recv got EOF from 127.0.0.1:51146 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 14:45:14.200085 19115 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:14.198983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.15.84.61:46423)
0810 14:45:14.199148 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:14.199152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:14.199270 (+ 118us) server_negotiation.cc:407] Connection header received
0810 14:45:14.199417 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:14.199419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:14.199472 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:14.199536 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:14.199955 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51160: BlockingRecv error: recv got EOF from 127.0.0.1:51160 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 14:45:14.267590 19115 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:14.266492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.15.84.61:46423)
0810 14:45:14.266655 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:14.266660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:14.266801 (+ 141us) server_negotiation.cc:407] Connection header received
0810 14:45:14.266930 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:14.266933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:14.266985 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:14.267056 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:14.267462 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51172: BlockingRecv error: recv got EOF from 127.0.0.1:51172 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 14:45:14.398967 19115 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:14.397895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.15.84.61:46423)
0810 14:45:14.398045 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:14.398050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:14.398197 (+ 147us) server_negotiation.cc:407] Connection header received
0810 14:45:14.398327 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:14.398329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:14.398379 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:14.398451 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:14.398842 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51184: BlockingRecv error: recv got EOF from 127.0.0.1:51184 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 14:45:14.658313 19115 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:45:14.657258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51186 (local address 127.15.84.61:46423)
0810 14:45:14.657401 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:14.657406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:14.657425 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:45:14.657612 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:14.657615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:14.657663 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:14.657735 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:45:14.658218 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51186: BlockingRecv error: recv got EOF from 127.0.0.1:51186 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:45:14.934319 19157 catalog_manager.cc:1259] Loaded cluster ID: aa3b5e5f95bc4789ac8f221d526e0014
I20260810 14:45:14.934412 19157 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a: loading cluster ID for follower catalog manager: success
I20260810 14:45:14.936231 19157 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a: acquiring CA information for follower catalog manager: success
I20260810 14:45:14.937566 19157 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 805fc62f4e2e407bae8798f417f50c4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:45:14.996430 19202 catalog_manager.cc:1259] Loaded cluster ID: aa3b5e5f95bc4789ac8f221d526e0014
I20260810 14:45:14.996529 19202 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8: loading cluster ID for follower catalog manager: success
I20260810 14:45:14.998377 19202 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8: acquiring CA information for follower catalog manager: success
I20260810 14:45:14.999703 19202 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50c08fbdc26848c4a26844ea8f5ae0b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:45:15.176613 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:15.172645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.15.84.61:46423)
0810 14:45:15.172901 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:15.172905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:15.172923 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:45:15.173052 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:15.173055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:15.173235 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:15.173349 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:45:15.173610 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:15.174155 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:15.175064 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:15.175312 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:15.175394 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:15.175687 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:45:15.176168 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:45:15.176269 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:15.176390 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 14:45:15.176449 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260810 14:45:15.184069 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:15.179968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.15.84.60:41059)
0810 14:45:15.180216 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:15.180220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:15.180234 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:45:15.180388 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:15.180391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:15.180524 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:15.180626 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:45:15.180855 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:15.181440 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:15.182427 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:15.182703 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:15.182798 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:15.183100 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:45:15.183598 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:45:15.183702 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:15.183822 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 14:45:15.183897 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":78,"threads_started":1}
I20260810 14:45:15.191694 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:15.187932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.15.84.60:41059)
0810 14:45:15.188055 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:15.188060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:15.188198 (+ 138us) server_negotiation.cc:407] Connection header received
0810 14:45:15.188343 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:15.188346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:15.188403 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:15.188469 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:45:15.188823 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:15.189462 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:15.190445 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:15.190713 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:15.190893 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:15.191171 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:45:15.191254 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:45:15.191298 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:15.191479 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 14:45:15.191547 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 14:45:15.192238 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:15.187494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.15.84.62:37257)
0810 14:45:15.187749 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:15.187754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:15.187988 (+ 234us) server_negotiation.cc:407] Connection header received
0810 14:45:15.188096 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:15.188100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:15.188274 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:15.188406 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:45:15.188673 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:15.189382 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:15.190661 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:15.190957 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:15.191062 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:15.191418 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:45:15.191821 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:45:15.191903 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:15.192029 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 14:45:15.192097 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":74,"threads_started":1}
I20260810 14:45:15.192930 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:45:15.187530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.15.84.61:46423)
0810 14:45:15.187675 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:45:15.187679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:45:15.188172 (+ 493us) server_negotiation.cc:407] Connection header received
0810 14:45:15.188284 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:45:15.188287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:45:15.188343 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:45:15.188411 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:45:15.189704 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:15.190490 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:15.191538 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:45:15.191864 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:45:15.191949 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:45:15.192455 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:45:15.192542 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:45:15.192625 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 14:45:15.192766 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 14:45:15.192830 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:45:15.195544 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 18773
I20260810 14:45:15.213351 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 18881
I20260810 14:45:15.227100 15696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPiaUUw/build/debug/bin/kudu with pid 18998
2026-08-10T14:45:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3278 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6524 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (295910 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST